Sep 15 22:27:06.174037 ip-10-0-138-116 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 22:27:06.595310 ip-10-0-138-116 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 22:27:06.595310 ip-10-0-138-116 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 22:27:06.595310 ip-10-0-138-116 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 22:27:06.595310 ip-10-0-138-116 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 22:27:06.595310 ip-10-0-138-116 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 22:27:06.596060 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.595401    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 22:27:06.599845 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599828    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 22:27:06.599845 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599842    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 22:27:06.599845 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599846    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599849    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599853    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599857    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599860    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599864    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599869    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599873    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599881    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599884    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599887    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599890    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599892    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599895    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599898    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599901    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599903    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599906    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599908    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599911    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 22:27:06.600027 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599914    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599917    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599920    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599923    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599926    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599931    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599934    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599937    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599940    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599943    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599946    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599948    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599951    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599954    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599957    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599959    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599962    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599965    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599967    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599970    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 22:27:06.600511 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599973    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599975    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599978    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599980    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599983    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599986    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599988    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599993    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.599997    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600000    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600003    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600005    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600008    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600011    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600013    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600016    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600018    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600021    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600024    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 22:27:06.601046 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600026    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600029    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600031    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600034    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600037    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600040    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600042    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600046    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600048    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600051    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600057    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600060    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600062    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600065    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600067    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600070    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600072    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600075    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600078    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600081    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 22:27:06.601500 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600084    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600086    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600089    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600091    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600094    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600097    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600099    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600102    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600104    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600106    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600109    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600111    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600114    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600116    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600118    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600121    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600123    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600126    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600841    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600848    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 22:27:06.602012 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600851    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600854    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600857    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600860    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600863    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600866    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600868    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600871    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600874    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600877    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600879    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600882    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600885    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600887    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600890    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600892    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600895    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600898    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600900    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600903    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 22:27:06.602498 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600905    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600908    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600911    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600913    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600916    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600919    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600921    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600923    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600926    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600928    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600931    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600934    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600937    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600941    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600945    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600948    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600951    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600954    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600957    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 22:27:06.603019 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600959    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600962    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600966    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600969    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600972    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600975    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600977    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600980    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600982    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600985    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600987    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600990    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600992    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600995    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.600999    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601002    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601005    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601008    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601011    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601014    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 22:27:06.603512 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601017    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601020    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601022    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601025    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601028    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601031    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601033    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601037    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601039    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601042    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601045    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601047    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601050    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601053    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601056    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601059    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601061    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601064    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601066    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 22:27:06.604033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601069    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601071    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601074    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601076    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601080    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601082    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601085    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601087    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601090    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601093    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601095    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601098    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601100    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601102    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601105    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601107    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601110    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601112    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601115    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601489    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 22:27:06.604503 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601494    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601497    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601501    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601506    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601509    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601512    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601516    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601519    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601521    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601524    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601526    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601529    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601531    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601534    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601536    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601539    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601542    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601545    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601547    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 22:27:06.604991 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601550    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601552    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601555    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601558    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601560    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601563    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601566    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601569    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601571    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601574    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601576    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601579    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601582    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601584    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601587    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601589    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601591    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601594    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601597    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601600    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 22:27:06.605462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601603    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601606    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601608    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601611    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601613    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601616    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601618    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601622    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601624    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601627    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601630    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601632    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601635    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601637    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601640    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601643    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601646    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601649    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601651    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601653    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 22:27:06.605963 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601656    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601659    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601661    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601664    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601666    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601669    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601672    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601674    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601677    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601680    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601684    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601687    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601689    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601693    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601696    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601699    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601702    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601705    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601707    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 22:27:06.606448 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601710    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601713    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601716    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601718    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601721    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601725    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601728    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601730    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601733    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601736    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601738    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601741    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601743    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601746    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601748    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601751    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601754    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601756    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601759    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.601761    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601844    2584 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 22:27:06.606919 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601885    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601892    2584 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601896    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601901    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601904    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601908    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601913    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601916    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601919    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601922    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601926    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601929    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601932    2584 flags.go:64] FLAG: --cgroup-root=""
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601937    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601940    2584 flags.go:64] FLAG: --client-ca-file=""
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601943    2584 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601946    2584 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601951    2584 flags.go:64] FLAG: --cluster-domain=""
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601953    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601957    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601960    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601963    2584 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601967    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 22:27:06.607432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601970    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601973    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601976    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601979    2584 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601982    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601985    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601988    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601991    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601996    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.601999    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602002    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602005    2584 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602007    2584 flags.go:64] FLAG: --enable-server="true"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602011    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602016    2584 flags.go:64] FLAG: --event-burst="100"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602019    2584 flags.go:64] FLAG: --event-qps="50"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602022    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602025    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602028    2584 flags.go:64] FLAG: --eviction-hard=""
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602032    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602034    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602037    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602042    2584 flags.go:64] FLAG: --eviction-soft=""
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602045    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602048    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 22:27:06.608004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602051    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602054    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602057    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602060    2584 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602062    2584 flags.go:64] FLAG: --feature-gates=""
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602066    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602069    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602072    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602075    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602078    2584 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602081    2584 flags.go:64] FLAG: --help="false"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602084    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-138-116.ec2.internal"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602087    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602091    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602094    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602098    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602101    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602104    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602107    2584 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602109    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602112    2584 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602116    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602119    2584 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602121    2584 flags.go:64] FLAG: --kube-reserved=""
Sep 15 22:27:06.608655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602124    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602127    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602130    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602132    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602135    2584 flags.go:64] FLAG: --lock-file=""
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602140    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602143    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602145    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602151    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602154    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602156    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602159    2584 flags.go:64] FLAG: --logging-format="text"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602162    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602166    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602169    2584 flags.go:64] FLAG: --manifest-url=""
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602171    2584 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602175    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602178    2584 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602182    2584 flags.go:64] FLAG: --max-pods="110"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602186    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602189    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602195    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602198    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602201    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602204    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 22:27:06.609247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602207    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602214    2584 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602217    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602220    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602223    2584 flags.go:64] FLAG: --pod-cidr=""
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602229    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602235    2584 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602238    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602241    2584 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602244    2584 flags.go:64] FLAG: --port="10250"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602247    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602250    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d202014ef8c00af0"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602253    2584 flags.go:64] FLAG: --qos-reserved=""
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602257    2584 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602260    2584 flags.go:64] FLAG: --register-node="true"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602263    2584 flags.go:64] FLAG: --register-with-taints=""
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602267    2584 flags.go:64] FLAG: --registry-burst="10"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602270    2584 flags.go:64] FLAG: --registry-qps="5"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602272    2584 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602275    2584 flags.go:64] FLAG: --reserved-memory=""
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602279    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602282    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602285    2584 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602288    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602291    2584 flags.go:64] FLAG: --runonce="false"
Sep 15 22:27:06.609875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602294    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602297    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602299    2584 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602304    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602307    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602310    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602313    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602316    2584 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602319    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602322    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602325    2584 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602328    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602330    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602334    2584 flags.go:64] FLAG: --system-cgroups=""
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602337    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602342    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602345    2584 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602347    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602352    2584 flags.go:64] FLAG: --tls-min-version=""
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602355    2584 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602359    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602362    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602365    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602367    2584 flags.go:64] FLAG: --v="2"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602371    2584 flags.go:64] FLAG: --version="false"
Sep 15 22:27:06.610465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602375    2584 flags.go:64] FLAG: --vmodule=""
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602381    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602384    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602501    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602505    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602508    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602511    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602514    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602517    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602519    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602523    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602526    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602529    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602531    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602534    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602537    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602539    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602542    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602545    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 22:27:06.611077 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602549    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602552    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602555    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602558    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602560    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602563    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602565    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602568    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602572    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602574    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602577    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602580    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602582    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602585    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602587    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602590    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602592    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602595    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602598    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602601    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 22:27:06.611764 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602603    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602606    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602608    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602612    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602615    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602617    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602620    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602622    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602625    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602628    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602630    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602633    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602636    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602638    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602641    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602644    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602647    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602649    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602652    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602654    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 22:27:06.612590 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602658    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602663    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602666    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602669    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602672    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602675    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602677    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602680    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602682    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602685    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602688    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602690    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602693    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602695    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602698    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602702    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602705    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602707    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602710    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 22:27:06.613230 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602712    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602715    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602718    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602720    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602723    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602725    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602728    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602730    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602733    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602736    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602739    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602741    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602744    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602747    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602751    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602755    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602758    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602761    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602763    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 22:27:06.613706 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602766    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602769    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602772    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602787    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.602790    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.602796    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.613166    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.613183    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.613197    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.613207    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613262    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613266    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613270    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613273    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613276    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 22:27:06.614197 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613279    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613282    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613285    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613288    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613291    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613294    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613297    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613300    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613304    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613307    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613310    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613313    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613316    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613325    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613327    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613330    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613334    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613336    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613339    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613342    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 22:27:06.614570 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613345    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613348    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613351    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613354    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613357    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613359    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613362    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613365    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613368    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613370    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613373    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613376    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613378    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613381    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613385    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613389    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613392    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613395    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613398    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613402    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 22:27:06.615069 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613404    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613408    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613410    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613413    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613416    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613419    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613422    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613424    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613427    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613430    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613432    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613435    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613438    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613440    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613443    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613445    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613448    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613451    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613453    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613456    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 22:27:06.615565 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613458    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613461    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613464    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613468    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613471    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613474    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613477    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613480    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613483    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613485    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613488    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613491    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613493    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613496    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613504    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613507    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613509    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613512    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613514    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 22:27:06.616075 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613517    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613520    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613522    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613525    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613528    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613530    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613533    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613536    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613538    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613541    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613545    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613549    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613552    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613554    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 22:27:06.616577 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613557    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.613562    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613673    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613678    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613681    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613684    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613687    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613690    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613693    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613695    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613699    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613703    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613706    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613710    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613719    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 22:27:06.616927 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613722    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613725    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613728    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613731    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613734    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613737    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613739    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613742    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613745    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613748    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613750    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613753    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613755    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613758    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613761    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613764    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613766    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613769    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613771    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613792    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 22:27:06.617289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613796    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613798    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613801    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613804    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613806    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613809    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613811    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613814    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613816    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613819    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613821    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613824    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613827    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613831    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613834    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613838    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613841    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613844    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613847    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613849    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 22:27:06.617770 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613852    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613855    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613858    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613861    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613864    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613867    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613869    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613872    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613875    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613878    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613880    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613883    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613886    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613888    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613891    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613893    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613896    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613898    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613901    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 22:27:06.618383 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613904    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613906    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613909    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613913    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613916    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613919    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613922    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613924    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613927    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613930    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613932    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613935    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613937    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613940    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613943    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613946    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613948    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613951    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613953    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613956    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 22:27:06.618865 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613958    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 22:27:06.619374 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613960    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 22:27:06.619374 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613963    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 22:27:06.619374 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613966    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 22:27:06.619374 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613968    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 22:27:06.619374 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613971    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 22:27:06.619374 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:06.613973    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 22:27:06.619374 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.613978    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 22:27:06.619374 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.614136    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 22:27:06.619374 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.618224    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 22:27:06.619374 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.619291    2584 server.go:1019] "Starting client certificate rotation"
Sep 15 22:27:06.619637 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.619391    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 22:27:06.619637 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.619424    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 22:27:06.622210 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.622197    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 22:27:06.623425 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.623408    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 22:27:06.640332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.640312    2584 log.go:25] "Validated CRI v1 runtime API"
Sep 15 22:27:06.644265 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.644248    2584 log.go:25] "Validated CRI v1 image API"
Sep 15 22:27:06.645110 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.645092    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 22:27:06.648550 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.648533    2584 fs.go:135] Filesystem UUIDs: map[47e5cd3b-1560-4515-bc4f-c03e8fcb7ffb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 df7d0b04-8275-4a6f-8f49-fa017f59768e:/dev/nvme0n1p4]
Sep 15 22:27:06.648616 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.648550    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 22:27:06.651752 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.651734    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 22:27:06.655645 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.655522    2584 manager.go:216] Machine: {Timestamp:2026-09-15 22:27:06.654107427 +0000 UTC m=+0.372412262 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099366 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec241c1bd971fcdd6ba844457f988f62 SystemUUID:ec241c1b-d971-fcdd-6ba8-44457f988f62 BootID:f5461e7f-2823-4b07-aa61-42e5763c4a63 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:f6:00:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:f6:00:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:09:e4:a0:29:94 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 22:27:06.655645 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.655632    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 22:27:06.655805 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.655724    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 22:27:06.659702 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.659671    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 22:27:06.659887 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.659703    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-116.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 22:27:06.660603 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.660591    2584 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 22:27:06.660638 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.660606    2584 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 22:27:06.660638 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.660613    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 22:27:06.661503 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.661492    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 22:27:06.662824 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.662814    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 22:27:06.667277 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.667266    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 22:27:06.667399 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.667390    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 22:27:06.667429 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.667418    2584 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 22:27:06.668510 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.668497    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 22:27:06.668548 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.668522    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 22:27:06.668548 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.668531    2584 kubelet.go:398] "Adding apiserver pod source"
Sep 15 22:27:06.668548 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.668540    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 22:27:06.668914 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.668903    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 22:27:06.668946 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.668922    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 22:27:06.671442 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.671426    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 22:27:06.674193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.674178    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 22:27:06.674280 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.674217    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 22:27:06.676390 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.676374    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 22:27:06.676477 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.676401    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 22:27:06.676477 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.676412    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 22:27:06.676477 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.676424    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 22:27:06.676477 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.676445    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 22:27:06.676477 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.676456    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 22:27:06.676477 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.676475    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 22:27:06.676663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.676483    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 22:27:06.676663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.676493    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 22:27:06.676663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.676501    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 22:27:06.676663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.676514    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 22:27:06.676663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.676523    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 22:27:06.677478 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.677458    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 22:27:06.677478 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.677476    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 22:27:06.678661 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.678628    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 22:27:06.678752 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.678652    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-116.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 22:27:06.681426 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.681261    2584 server.go:1268] "Started kubelet"
Sep 15 22:27:06.681496 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.681328    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 22:27:06.681563 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.681350    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 22:27:06.681677 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.681583    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 22:27:06.683198 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.683182    2584 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 22:27:06.683304 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.683287    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 22:27:06.683821 ip-10-0-138-116 systemd[1]: Started Kubernetes Kubelet.
Sep 15 22:27:06.686324 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.685666    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 22:27:06.687961 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.687940    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-116.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 22:27:06.689135 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.687953    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-116.ec2.internal.18d59efabbe89307  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-116.ec2.internal,UID:ip-10-0-138-116.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-116.ec2.internal,},FirstTimestamp:2026-09-15 22:27:06.681234183 +0000 UTC m=+0.399539019,LastTimestamp:2026-09-15 22:27:06.681234183 +0000 UTC m=+0.399539019,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-116.ec2.internal,}"
Sep 15 22:27:06.691149 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.691127    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 22:27:06.691750 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.691736    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 22:27:06.691899 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.691879    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-116.ec2.internal\" not found"
Sep 15 22:27:06.692620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.692591    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 22:27:06.692620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.692593    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 22:27:06.692620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.692621    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 22:27:06.692814 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.692697    2584 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 22:27:06.692814 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.692710    2584 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 22:27:06.693446 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.693430    2584 factory.go:55] Registering systemd factory
Sep 15 22:27:06.693532 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.693465    2584 factory.go:223] Registration of the systemd container factory successfully
Sep 15 22:27:06.693658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.693645    2584 factory.go:153] Registering CRI-O factory
Sep 15 22:27:06.693694 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.693661    2584 factory.go:223] Registration of the crio container factory successfully
Sep 15 22:27:06.693734 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.693711    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 22:27:06.693734 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.693729    2584 factory.go:103] Registering Raw factory
Sep 15 22:27:06.693844 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.693740    2584 manager.go:1196] Started watching for new ooms in manager
Sep 15 22:27:06.694961 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.694938    2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-116.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 22:27:06.695088 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.695059    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 22:27:06.695154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.695117    2584 manager.go:349] Starting recovery of all containers
Sep 15 22:27:06.707243 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.707132    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pbmph"
Sep 15 22:27:06.707631 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.707617    2584 manager.go:354] Recovery completed
Sep 15 22:27:06.711703 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.711690    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 22:27:06.713547 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.713532    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 22:27:06.713636 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.713558    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 22:27:06.713636 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.713572    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientPID"
Sep 15 22:27:06.714109 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.714094    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 22:27:06.714109 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.714107    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 22:27:06.714207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.714123    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 22:27:06.714207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.714108    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 22:27:06.714740 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.714724    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pbmph"
Sep 15 22:27:06.714951 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.714934    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 22:27:06.715009 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.714962    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 22:27:06.715009 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.714972    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientPID"
Sep 15 22:27:06.715704 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.715627    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-116.ec2.internal.18d59efabdd59f00  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-116.ec2.internal,UID:ip-10-0-138-116.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-116.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-116.ec2.internal,},FirstTimestamp:2026-09-15 22:27:06.713546496 +0000 UTC m=+0.431851332,LastTimestamp:2026-09-15 22:27:06.713546496 +0000 UTC m=+0.431851332,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-116.ec2.internal,}"
Sep 15 22:27:06.716148 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.716133    2584 policy_none.go:49] "None policy: Start"
Sep 15 22:27:06.716148 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.716150    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 22:27:06.716249 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.716161    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 22:27:06.717972 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.717960    2584 policy_none.go:47] "Start"
Sep 15 22:27:06.738343 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.738330    2584 manager.go:337] "Starting Device Plugin manager"
Sep 15 22:27:06.738411 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.738369    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 22:27:06.738411 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.738381    2584 server.go:87] "Starting device plugin registration server"
Sep 15 22:27:06.738667 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.738656    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 22:27:06.738708 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.738672    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 22:27:06.738846 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.738828    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 22:27:06.738925 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.738898    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 22:27:06.738925 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.738906    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 22:27:06.739372 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.739350    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 22:27:06.739459 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.739407    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-116.ec2.internal\" not found"
Sep 15 22:27:06.791497 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.791467    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 22:27:06.793575 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.792551    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 22:27:06.793575 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.792590    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 22:27:06.793575 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.792613    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 22:27:06.793575 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.792671    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 22:27:06.794699 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.794681    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 22:27:06.839304 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.839285    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 22:27:06.840091 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.840074    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 22:27:06.840171 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.840103    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 22:27:06.840171 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.840114    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientPID"
Sep 15 22:27:06.840171 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.840134    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-116.ec2.internal"
Sep 15 22:27:06.850072 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.850009    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-116.ec2.internal"
Sep 15 22:27:06.850072 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.850029    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-116.ec2.internal\": node \"ip-10-0-138-116.ec2.internal\" not found"
Sep 15 22:27:06.873449 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.873428    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-116.ec2.internal\" not found"
Sep 15 22:27:06.893561 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.893536    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-116.ec2.internal"]
Sep 15 22:27:06.893628 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.893600    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 22:27:06.895054 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.895032    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 22:27:06.895148 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.895061    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 22:27:06.895148 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.895075    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientPID"
Sep 15 22:27:06.897417 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.897401    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 22:27:06.897528 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.897511    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:06.897584 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.897553    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 22:27:06.898147 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.898124    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 22:27:06.898147 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.898126    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 22:27:06.898288 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.898171    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 22:27:06.898288 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.898183    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientPID"
Sep 15 22:27:06.898288 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.898149    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 22:27:06.898288 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.898232    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientPID"
Sep 15 22:27:06.900412 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.900393    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:06.900498 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.900421    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 22:27:06.901065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.901053    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 22:27:06.901129 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.901075    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 22:27:06.901129 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.901084    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeHasSufficientPID"
Sep 15 22:27:06.928388 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.928374    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-116.ec2.internal\" not found" node="ip-10-0-138-116.ec2.internal"
Sep 15 22:27:06.932944 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.932913    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-116.ec2.internal\" not found" node="ip-10-0-138-116.ec2.internal"
Sep 15 22:27:06.973499 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:06.973472    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-116.ec2.internal\" not found"
Sep 15 22:27:06.993763 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.993745    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df62c586d424be3422da26b000edf8eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal\" (UID: \"df62c586d424be3422da26b000edf8eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:06.993838 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.993768    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/69eda65fcd513f716093f5dd7f2f0e50-config\") pod \"kube-apiserver-proxy-ip-10-0-138-116.ec2.internal\" (UID: \"69eda65fcd513f716093f5dd7f2f0e50\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:06.993838 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:06.993796    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/df62c586d424be3422da26b000edf8eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal\" (UID: \"df62c586d424be3422da26b000edf8eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:07.073952 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:07.073919    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-116.ec2.internal\" not found"
Sep 15 22:27:07.094298 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:07.094258    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/df62c586d424be3422da26b000edf8eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal\" (UID: \"df62c586d424be3422da26b000edf8eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:07.094366 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:07.094304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df62c586d424be3422da26b000edf8eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal\" (UID: \"df62c586d424be3422da26b000edf8eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:07.094366 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:07.094324    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/69eda65fcd513f716093f5dd7f2f0e50-config\") pod \"kube-apiserver-proxy-ip-10-0-138-116.ec2.internal\" (UID: \"69eda65fcd513f716093f5dd7f2f0e50\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:07.094366 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:07.094348    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/df62c586d424be3422da26b000edf8eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal\" (UID: \"df62c586d424be3422da26b000edf8eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:07.094366 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:07.094357    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df62c586d424be3422da26b000edf8eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal\" (UID: \"df62c586d424be3422da26b000edf8eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:07.094490 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:07.094357    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/69eda65fcd513f716093f5dd7f2f0e50-config\") pod \"kube-apiserver-proxy-ip-10-0-138-116.ec2.internal\" (UID: \"69eda65fcd513f716093f5dd7f2f0e50\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:07.174805 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:07.174718    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-116.ec2.internal\" not found"
Sep 15 22:27:07.232193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:07.232161    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:07.237606 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:07.237336    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:07.275373 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:07.275350    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-116.ec2.internal\" not found"
Sep 15 22:27:07.375937 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:07.375906    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-116.ec2.internal\" not found"
Sep 15 22:27:08.575959 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:08.575936    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-116.ec2.internal\" not found"
Sep 15 22:27:08.663241 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:08.663214    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 22:27:08.676552 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:08.676532    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-116.ec2.internal\" not found"
Sep 15 22:27:08.717997 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:08.717975    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 22:27:08.718394 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:08.718089    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 22:27:08.718394 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:08.718110    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 22:27:08.777505 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:08.777482    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-116.ec2.internal\" not found"
Sep 15 22:27:08.820400 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:08.820331    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 22:22:07 +0000 UTC" deadline="2028-02-20 20:52:20.38796135 +0000 UTC"
Sep 15 22:27:08.820400 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:08.820358    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12550h25m11.567606874s"
Sep 15 22:27:08.877602 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:08.877582    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-116.ec2.internal\" not found"
Sep 15 22:27:08.946654 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:08.946631    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 22:27:08.992244 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:08.992220    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:09.005020 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.004884    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 22:27:09.005836 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.005823    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal"
Sep 15 22:27:09.011527 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:09.011498    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69eda65fcd513f716093f5dd7f2f0e50.slice/crio-722605015de7a6004a8fbf9fdfb4ef3afe591c94477aaa7d3376c4084758461f WatchSource:0}: Error finding container 722605015de7a6004a8fbf9fdfb4ef3afe591c94477aaa7d3376c4084758461f: Status 404 returned error can't find the container with id 722605015de7a6004a8fbf9fdfb4ef3afe591c94477aaa7d3376c4084758461f
Sep 15 22:27:09.016106 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.016091    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 22:27:09.023755 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.023739    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 22:27:09.039899 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:09.039870    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf62c586d424be3422da26b000edf8eb.slice/crio-d4ffaf6e0c3f8b84da04f54738c9ce41c1c1035141cb7636df0555dc39df7cd5 WatchSource:0}: Error finding container d4ffaf6e0c3f8b84da04f54738c9ce41c1c1035141cb7636df0555dc39df7cd5: Status 404 returned error can't find the container with id d4ffaf6e0c3f8b84da04f54738c9ce41c1c1035141cb7636df0555dc39df7cd5
Sep 15 22:27:09.167920 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.167862    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 22:27:09.768360 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.768327    2584 apiserver.go:52] "Watching apiserver"
Sep 15 22:27:09.777311 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.777285    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 22:27:09.777776 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.777753    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-hd5sn","kube-system/konnectivity-agent-5nc6l","openshift-cluster-node-tuning-operator/tuned-q72l4","openshift-image-registry/node-ca-8jwbj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal","openshift-multus/multus-additional-cni-plugins-5fz5x","openshift-multus/multus-llx9t","openshift-network-diagnostics/network-check-target-jc5qb","openshift-network-operator/iptables-alerter-xzzl6","kube-system/global-pull-secret-syncer-mn2ls","kube-system/kube-apiserver-proxy-ip-10-0-138-116.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z","openshift-dns/node-resolver-dwfjr","openshift-multus/network-metrics-daemon-pc4cx"]
Sep 15 22:27:09.780288 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.780256    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:09.780378 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:09.780348    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:09.783207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.782638    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5nc6l"
Sep 15 22:27:09.785283 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.785243    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 22:27:09.785645 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.785518    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7bpdd\""
Sep 15 22:27:09.785809 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.785770    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 22:27:09.787641 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.787077    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.789317 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.789282    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8jwbj"
Sep 15 22:27:09.789940 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.789919    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vxqpm\""
Sep 15 22:27:09.789940 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.789938    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 22:27:09.790074 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.790036    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 22:27:09.790129 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.790075    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 22:27:09.790198 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.790175    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 22:27:09.792005 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.791988    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 22:27:09.792721 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.792703    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.793329 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.793307    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 22:27:09.793418 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.793372    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqbx5\""
Sep 15 22:27:09.793477 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.793412    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 22:27:09.794074 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.793647    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 22:27:09.794074 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.793696    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.795341 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.795323    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 22:27:09.795556 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.795540    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 22:27:09.795990 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.795864    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hmjfv\""
Sep 15 22:27:09.797357 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.796318    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 22:27:09.797357 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.796607    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:09.797357 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:09.796670    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:09.797357 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.796934    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 22:27:09.798037 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.798019    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pgtcl\""
Sep 15 22:27:09.798116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.798060    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 22:27:09.798454 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.798255    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 22:27:09.798454 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.798357    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 22:27:09.798454 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.798256    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 22:27:09.799289 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.798963    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzzl6"
Sep 15 22:27:09.801514 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801352    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-cnibin\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.801514 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801395    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-run-multus-certs\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.801514 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801422    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgm72\" (UniqueName: \"kubernetes.io/projected/c298fcdd-c5f5-4410-995a-8d66952752b8-kube-api-access-mgm72\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.801514 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801447    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c8e67238-6768-4b3c-89f2-a17e61edee60-serviceca\") pod \"node-ca-8jwbj\" (UID: \"c8e67238-6768-4b3c-89f2-a17e61edee60\") " pod="openshift-image-registry/node-ca-8jwbj"
Sep 15 22:27:09.801514 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801494    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.801804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801526    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-systemd-units\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.801804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801549    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-slash\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.801804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801575    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-var-lib-cni-multus\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.801804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801598    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.801804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801619    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-node-log\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.801804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801641    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5e931f99-26f8-440d-a9b9-269517778506-ovnkube-script-lib\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.801804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801700    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38aa201f-97b9-443e-b246-dd19d606eb43-agent-certs\") pod \"konnectivity-agent-5nc6l\" (UID: \"38aa201f-97b9-443e-b246-dd19d606eb43\") " pod="kube-system/konnectivity-agent-5nc6l"
Sep 15 22:27:09.801804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801743    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38aa201f-97b9-443e-b246-dd19d606eb43-konnectivity-ca\") pod \"konnectivity-agent-5nc6l\" (UID: \"38aa201f-97b9-443e-b246-dd19d606eb43\") " pod="kube-system/konnectivity-agent-5nc6l"
Sep 15 22:27:09.801804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801773    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-system-cni-dir\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.801804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801804    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-multus-cni-dir\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801822    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c298fcdd-c5f5-4410-995a-8d66952752b8-cni-binary-copy\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801872    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-var-lib-kubelet\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801901    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-multus-conf-dir\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801928    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldgx5\" (UniqueName: \"kubernetes.io/projected/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-kube-api-access-ldgx5\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801949    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:09.802023    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.801953    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs\") pod \"network-metrics-daemon-pc4cx\" (UID: \"e8d2b3b9-6751-4c72-9677-dabc86c07f2a\") " pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802062    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrp82\" (UniqueName: \"kubernetes.io/projected/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-kube-api-access-wrp82\") pod \"network-metrics-daemon-pc4cx\" (UID: \"e8d2b3b9-6751-4c72-9677-dabc86c07f2a\") " pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802096    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-run-k8s-cni-cncf-io\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802119    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-var-lib-cni-bin\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802147    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-hostroot\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802163    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-run-openvswitch\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802176    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-log-socket\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802188    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-cni-bin\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802213    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-cnibin\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bwcs\" (UniqueName: \"kubernetes.io/projected/c8e67238-6768-4b3c-89f2-a17e61edee60-kube-api-access-7bwcs\") pod \"node-ca-8jwbj\" (UID: \"c8e67238-6768-4b3c-89f2-a17e61edee60\") " pod="openshift-image-registry/node-ca-8jwbj"
Sep 15 22:27:09.802378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802292    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-system-cni-dir\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.803193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802328    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-cni-binary-copy\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.803193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802351    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.803193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802385    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-var-lib-openvswitch\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.803193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802405    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-cni-netd\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.803193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802425    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-run-netns\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.803193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802448    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-etc-openvswitch\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.803193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802490    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-run-ovn\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.803193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802523    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5e931f99-26f8-440d-a9b9-269517778506-ovnkube-config\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.803193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802548    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv56l\" (UniqueName: \"kubernetes.io/projected/5e931f99-26f8-440d-a9b9-269517778506-kube-api-access-bv56l\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.803193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802570    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-etc-kubernetes\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.803193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802605    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-kubelet\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.803193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802635    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-run-netns\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.805539 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.804634    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 22:27:09.805539 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.804675    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 22:27:09.805539 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.804681    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 22:27:09.805539 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.802658    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-run-systemd\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.805539 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.805238    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-run-ovn-kubernetes\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.805539 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.805297    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7m7v6\""
Sep 15 22:27:09.805539 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.805298    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.805539 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.805504    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5e931f99-26f8-440d-a9b9-269517778506-env-overrides\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.805539 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.805536    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e931f99-26f8-440d-a9b9-269517778506-ovn-node-metrics-cert\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.805978 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.805567    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-multus-socket-dir-parent\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.805978 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.805595    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c298fcdd-c5f5-4410-995a-8d66952752b8-multus-daemon-config\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.805978 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.805623    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8e67238-6768-4b3c-89f2-a17e61edee60-host\") pod \"node-ca-8jwbj\" (UID: \"c8e67238-6768-4b3c-89f2-a17e61edee60\") " pod="openshift-image-registry/node-ca-8jwbj"
Sep 15 22:27:09.807357 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.807280    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:09.808932 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.808914    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 22:27:09.809755 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.809657    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7ssbg\""
Sep 15 22:27:09.809954 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.809916    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 22:27:09.810158 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.810128    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 22:27:09.810715 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.810695    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 22:27:09.810997 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.810978    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dwfjr"
Sep 15 22:27:09.813531 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.813488    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mzx62\""
Sep 15 22:27:09.813785 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.813770    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 22:27:09.813981 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.813965    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.814044 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.814011    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 22:27:09.816880 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.816859    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 22:27:09.817000 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.816916    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ldz54\""
Sep 15 22:27:09.817085 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.817072    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 22:27:09.832608 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.832588    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pp5vc"
Sep 15 22:27:09.838017 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.837998    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pp5vc"
Sep 15 22:27:09.894123 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.894097    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 22:27:09.898150 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.898103    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal" event={"ID":"df62c586d424be3422da26b000edf8eb","Type":"ContainerStarted","Data":"d4ffaf6e0c3f8b84da04f54738c9ce41c1c1035141cb7636df0555dc39df7cd5"}
Sep 15 22:27:09.899567 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.899544    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-116.ec2.internal" event={"ID":"69eda65fcd513f716093f5dd7f2f0e50","Type":"ContainerStarted","Data":"722605015de7a6004a8fbf9fdfb4ef3afe591c94477aaa7d3376c4084758461f"}
Sep 15 22:27:09.906909 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.906764    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-etc-kubernetes\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.906909 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.906801    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-kubelet\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.906909 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.906825    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-run-netns\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.906909 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.906849    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-run-systemd\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.906909 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.906873    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-run-ovn-kubernetes\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.906910    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-kubelet\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.906921    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9br\" (UniqueName: \"kubernetes.io/projected/e3deae5a-e1be-4af6-8ac0-ef4976250510-kube-api-access-gr9br\") pod \"node-resolver-dwfjr\" (UID: \"e3deae5a-e1be-4af6-8ac0-ef4976250510\") " pod="openshift-dns/node-resolver-dwfjr"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.906932    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-etc-kubernetes\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.906912    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-run-netns\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.906960    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-multus-socket-dir-parent\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.906927    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-run-systemd\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.906989    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8e67238-6768-4b3c-89f2-a17e61edee60-host\") pod \"node-ca-8jwbj\" (UID: \"c8e67238-6768-4b3c-89f2-a17e61edee60\") " pod="openshift-image-registry/node-ca-8jwbj"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.906990    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-run-ovn-kubernetes\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907023    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdf53f14-911c-4dac-94d9-caf371cfc501-tmp\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907090    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-cnibin\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907101    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8e67238-6768-4b3c-89f2-a17e61edee60-host\") pod \"node-ca-8jwbj\" (UID: \"c8e67238-6768-4b3c-89f2-a17e61edee60\") " pod="openshift-image-registry/node-ca-8jwbj"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907125    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-multus-socket-dir-parent\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907137    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-run-multus-certs\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907164    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgm72\" (UniqueName: \"kubernetes.io/projected/c298fcdd-c5f5-4410-995a-8d66952752b8-kube-api-access-mgm72\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.907207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907175    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-cnibin\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907217    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907242    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-node-log\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907326    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-run-multus-certs\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907332    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-node-log\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907391    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c91b9f47-6ec9-47fe-b514-d4a26101966b-iptables-alerter-script\") pod \"iptables-alerter-xzzl6\" (UID: \"c91b9f47-6ec9-47fe-b514-d4a26101966b\") " pod="openshift-network-operator/iptables-alerter-xzzl6"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907427    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907461    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr8ds\" (UniqueName: \"kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds\") pod \"network-check-target-jc5qb\" (UID: \"752e7fb0-13a9-4aee-b255-eb0eb8c04bdf\") " pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907486    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-multus-cni-dir\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907520    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-var-lib-kubelet\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907564    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-multus-cni-dir\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907601    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldgx5\" (UniqueName: \"kubernetes.io/projected/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-kube-api-access-ldgx5\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907633    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-var-lib-kubelet\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907671    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-log-socket\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907779    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-systemd\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907809    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-log-socket\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907810    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-sys\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.907943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907824    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907845    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs\") pod \"network-metrics-daemon-pc4cx\" (UID: \"e8d2b3b9-6751-4c72-9677-dabc86c07f2a\") " pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907869    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-var-lib-cni-bin\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907902    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-hostroot\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907925    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-sys-fs\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907947    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-var-lib-cni-bin\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907963    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3deae5a-e1be-4af6-8ac0-ef4976250510-tmp-dir\") pod \"node-resolver-dwfjr\" (UID: \"e3deae5a-e1be-4af6-8ac0-ef4976250510\") " pod="openshift-dns/node-resolver-dwfjr"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:09.907970    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.907989    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-sysctl-conf\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908001    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-hostroot\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908010    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5e931f99-26f8-440d-a9b9-269517778506-ovnkube-config\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:09.908029    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs podName:e8d2b3b9-6751-4c72-9677-dabc86c07f2a nodeName:}" failed. No retries permitted until 2026-09-15 22:27:10.408009226 +0000 UTC m=+3.026824130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs") pod "network-metrics-daemon-pc4cx" (UID: "e8d2b3b9-6751-4c72-9677-dabc86c07f2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908124    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv56l\" (UniqueName: \"kubernetes.io/projected/5e931f99-26f8-440d-a9b9-269517778506-kube-api-access-bv56l\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908155    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-cnibin\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908181    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bwcs\" (UniqueName: \"kubernetes.io/projected/c8e67238-6768-4b3c-89f2-a17e61edee60-kube-api-access-7bwcs\") pod \"node-ca-8jwbj\" (UID: \"c8e67238-6768-4b3c-89f2-a17e61edee60\") " pod="openshift-image-registry/node-ca-8jwbj"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908240    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-cnibin\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908303    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-tuned\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.908705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908331    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-run-netns\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908369    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-run-ovn\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908404    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-etc-selinux\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908443    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-run-ovn\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908446    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908492    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908434    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-run-netns\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908495    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e931f99-26f8-440d-a9b9-269517778506-ovn-node-metrics-cert\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908611    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-kubernetes\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908666    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-lib-modules\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908691    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-var-lib-kubelet\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908734    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c298fcdd-c5f5-4410-995a-8d66952752b8-multus-daemon-config\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908758    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-registration-dir\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908783    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftsqr\" (UniqueName: \"kubernetes.io/projected/c91b9f47-6ec9-47fe-b514-d4a26101966b-kube-api-access-ftsqr\") pod \"iptables-alerter-xzzl6\" (UID: \"c91b9f47-6ec9-47fe-b514-d4a26101966b\") " pod="openshift-network-operator/iptables-alerter-xzzl6"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908804    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-sysctl-d\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908829    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c8e67238-6768-4b3c-89f2-a17e61edee60-serviceca\") pod \"node-ca-8jwbj\" (UID: \"c8e67238-6768-4b3c-89f2-a17e61edee60\") " pod="openshift-image-registry/node-ca-8jwbj"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908851    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-systemd-units\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.909253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-slash\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908899    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5e931f99-26f8-440d-a9b9-269517778506-ovnkube-script-lib\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908926    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-socket-dir\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908947    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-run\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908973    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-host\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.908997    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-var-lib-cni-multus\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909022    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909059    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-kubelet-config\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909081    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-modprobe-d\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909105    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909109    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909136    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38aa201f-97b9-443e-b246-dd19d606eb43-agent-certs\") pod \"konnectivity-agent-5nc6l\" (UID: \"38aa201f-97b9-443e-b246-dd19d606eb43\") " pod="kube-system/konnectivity-agent-5nc6l"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909165    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38aa201f-97b9-443e-b246-dd19d606eb43-konnectivity-ca\") pod \"konnectivity-agent-5nc6l\" (UID: \"38aa201f-97b9-443e-b246-dd19d606eb43\") " pod="kube-system/konnectivity-agent-5nc6l"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909187    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-system-cni-dir\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909226    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c298fcdd-c5f5-4410-995a-8d66952752b8-cni-binary-copy\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-multus-conf-dir\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909286    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c298fcdd-c5f5-4410-995a-8d66952752b8-multus-daemon-config\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.909933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909306    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwql5\" (UniqueName: \"kubernetes.io/projected/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-kube-api-access-nwql5\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909331    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-sysconfig\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909362    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-var-lib-cni-multus\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909395    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5e931f99-26f8-440d-a9b9-269517778506-env-overrides\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909512    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5e931f99-26f8-440d-a9b9-269517778506-ovnkube-config\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909521    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-system-cni-dir\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909707    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-systemd-units\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909910    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909928    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c298fcdd-c5f5-4410-995a-8d66952752b8-cni-binary-copy\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.909984    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-multus-conf-dir\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910043    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrp82\" (UniqueName: \"kubernetes.io/projected/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-kube-api-access-wrp82\") pod \"network-metrics-daemon-pc4cx\" (UID: \"e8d2b3b9-6751-4c72-9677-dabc86c07f2a\") " pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910052    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38aa201f-97b9-443e-b246-dd19d606eb43-konnectivity-ca\") pod \"konnectivity-agent-5nc6l\" (UID: \"38aa201f-97b9-443e-b246-dd19d606eb43\") " pod="kube-system/konnectivity-agent-5nc6l"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910119    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-run-k8s-cni-cncf-io\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910141    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c8e67238-6768-4b3c-89f2-a17e61edee60-serviceca\") pod \"node-ca-8jwbj\" (UID: \"c8e67238-6768-4b3c-89f2-a17e61edee60\") " pod="openshift-image-registry/node-ca-8jwbj"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910245    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5e931f99-26f8-440d-a9b9-269517778506-env-overrides\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910282    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-run-openvswitch\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5e931f99-26f8-440d-a9b9-269517778506-ovnkube-script-lib\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910306    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c298fcdd-c5f5-4410-995a-8d66952752b8-host-run-k8s-cni-cncf-io\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.910662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910330    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-run-openvswitch\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910334    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-slash\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910357    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-cni-bin\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910392    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-cni-bin\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910426    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-cni-netd\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910463    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-device-dir\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910489    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-host-cni-netd\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910488    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3deae5a-e1be-4af6-8ac0-ef4976250510-hosts-file\") pod \"node-resolver-dwfjr\" (UID: \"e3deae5a-e1be-4af6-8ac0-ef4976250510\") " pod="openshift-dns/node-resolver-dwfjr"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910539    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-system-cni-dir\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910565    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-cni-binary-copy\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910589    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910600    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-system-cni-dir\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910614    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-var-lib-openvswitch\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910653    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c91b9f47-6ec9-47fe-b514-d4a26101966b-host-slash\") pod \"iptables-alerter-xzzl6\" (UID: \"c91b9f47-6ec9-47fe-b514-d4a26101966b\") " pod="openshift-network-operator/iptables-alerter-xzzl6"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910677    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-dbus\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-etc-openvswitch\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910714    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-var-lib-openvswitch\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.911565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910724    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7twqw\" (UniqueName: \"kubernetes.io/projected/cdf53f14-911c-4dac-94d9-caf371cfc501-kube-api-access-7twqw\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:09.912393 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910748    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.912393 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.910796    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5e931f99-26f8-440d-a9b9-269517778506-etc-openvswitch\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.912393 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.911025    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-cni-binary-copy\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.913994 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.913951    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38aa201f-97b9-443e-b246-dd19d606eb43-agent-certs\") pod \"konnectivity-agent-5nc6l\" (UID: \"38aa201f-97b9-443e-b246-dd19d606eb43\") " pod="kube-system/konnectivity-agent-5nc6l"
Sep 15 22:27:09.914471 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.914437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e931f99-26f8-440d-a9b9-269517778506-ovn-node-metrics-cert\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:09.920413 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.919625    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bwcs\" (UniqueName: \"kubernetes.io/projected/c8e67238-6768-4b3c-89f2-a17e61edee60-kube-api-access-7bwcs\") pod \"node-ca-8jwbj\" (UID: \"c8e67238-6768-4b3c-89f2-a17e61edee60\") " pod="openshift-image-registry/node-ca-8jwbj"
Sep 15 22:27:09.920413 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.919652    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldgx5\" (UniqueName: \"kubernetes.io/projected/86c5e1a3-16e0-4171-a9a2-df5f8eba5318-kube-api-access-ldgx5\") pod \"multus-additional-cni-plugins-5fz5x\" (UID: \"86c5e1a3-16e0-4171-a9a2-df5f8eba5318\") " pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:09.920413 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.920212    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgm72\" (UniqueName: \"kubernetes.io/projected/c298fcdd-c5f5-4410-995a-8d66952752b8-kube-api-access-mgm72\") pod \"multus-llx9t\" (UID: \"c298fcdd-c5f5-4410-995a-8d66952752b8\") " pod="openshift-multus/multus-llx9t"
Sep 15 22:27:09.921906 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.921879    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrp82\" (UniqueName: \"kubernetes.io/projected/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-kube-api-access-wrp82\") pod \"network-metrics-daemon-pc4cx\" (UID: \"e8d2b3b9-6751-4c72-9677-dabc86c07f2a\") " pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:09.922829 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:09.922805    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv56l\" (UniqueName: \"kubernetes.io/projected/5e931f99-26f8-440d-a9b9-269517778506-kube-api-access-bv56l\") pod \"ovnkube-node-hd5sn\" (UID: \"5e931f99-26f8-440d-a9b9-269517778506\") " pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:10.011317 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011289    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-tuned\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.011508 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011325    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-etc-selinux\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.011508 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011349    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-kubernetes\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.011508 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011412    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-kubernetes\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.011508 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011444    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-lib-modules\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.011508 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011468    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-var-lib-kubelet\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.011508 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011470    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-etc-selinux\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.011508 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011489    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-registration-dir\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011512    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftsqr\" (UniqueName: \"kubernetes.io/projected/c91b9f47-6ec9-47fe-b514-d4a26101966b-kube-api-access-ftsqr\") pod \"iptables-alerter-xzzl6\" (UID: \"c91b9f47-6ec9-47fe-b514-d4a26101966b\") " pod="openshift-network-operator/iptables-alerter-xzzl6"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011534    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-sysctl-d\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011537    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-var-lib-kubelet\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011562    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-socket-dir\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011571    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-lib-modules\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011585    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-run\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011584    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-registration-dir\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011640    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-run\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011683    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-sysctl-d\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-host\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011702    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-socket-dir\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011753    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-host\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011807    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-kubelet-config\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011856    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-kubelet-config\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:10.011858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011866    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-modprobe-d\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011892    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011922    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwql5\" (UniqueName: \"kubernetes.io/projected/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-kube-api-access-nwql5\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011935    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-modprobe-d\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011945    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-sysconfig\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-device-dir\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011985    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.011994    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3deae5a-e1be-4af6-8ac0-ef4976250510-hosts-file\") pod \"node-resolver-dwfjr\" (UID: \"e3deae5a-e1be-4af6-8ac0-ef4976250510\") " pod="openshift-dns/node-resolver-dwfjr"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012031    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3deae5a-e1be-4af6-8ac0-ef4976250510-hosts-file\") pod \"node-resolver-dwfjr\" (UID: \"e3deae5a-e1be-4af6-8ac0-ef4976250510\") " pod="openshift-dns/node-resolver-dwfjr"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012040    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c91b9f47-6ec9-47fe-b514-d4a26101966b-host-slash\") pod \"iptables-alerter-xzzl6\" (UID: \"c91b9f47-6ec9-47fe-b514-d4a26101966b\") " pod="openshift-network-operator/iptables-alerter-xzzl6"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012080    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-sysconfig\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-dbus\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012121    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-device-dir\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012082    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c91b9f47-6ec9-47fe-b514-d4a26101966b-host-slash\") pod \"iptables-alerter-xzzl6\" (UID: \"c91b9f47-6ec9-47fe-b514-d4a26101966b\") " pod="openshift-network-operator/iptables-alerter-xzzl6"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7twqw\" (UniqueName: \"kubernetes.io/projected/cdf53f14-911c-4dac-94d9-caf371cfc501-kube-api-access-7twqw\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012168    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9br\" (UniqueName: \"kubernetes.io/projected/e3deae5a-e1be-4af6-8ac0-ef4976250510-kube-api-access-gr9br\") pod \"node-resolver-dwfjr\" (UID: \"e3deae5a-e1be-4af6-8ac0-ef4976250510\") " pod="openshift-dns/node-resolver-dwfjr"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012193    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdf53f14-911c-4dac-94d9-caf371cfc501-tmp\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.012585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012227    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c91b9f47-6ec9-47fe-b514-d4a26101966b-iptables-alerter-script\") pod \"iptables-alerter-xzzl6\" (UID: \"c91b9f47-6ec9-47fe-b514-d4a26101966b\") " pod="openshift-network-operator/iptables-alerter-xzzl6"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012246    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-dbus\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8ds\" (UniqueName: \"kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds\") pod \"network-check-target-jc5qb\" (UID: \"752e7fb0-13a9-4aee-b255-eb0eb8c04bdf\") " pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012335    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-systemd\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.012346    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012357    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-sys\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012401    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-sys-fs\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.012418    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret podName:7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c nodeName:}" failed. No retries permitted until 2026-09-15 22:27:10.512399835 +0000 UTC m=+3.131214734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret") pod "global-pull-secret-syncer-mn2ls" (UID: "7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012445    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-sys-fs\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012448    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3deae5a-e1be-4af6-8ac0-ef4976250510-tmp-dir\") pod \"node-resolver-dwfjr\" (UID: \"e3deae5a-e1be-4af6-8ac0-ef4976250510\") " pod="openshift-dns/node-resolver-dwfjr"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012474    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-sysctl-conf\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012615    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-sysctl-conf\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012620    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-systemd\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012665    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdf53f14-911c-4dac-94d9-caf371cfc501-sys\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.012986    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3deae5a-e1be-4af6-8ac0-ef4976250510-tmp-dir\") pod \"node-resolver-dwfjr\" (UID: \"e3deae5a-e1be-4af6-8ac0-ef4976250510\") " pod="openshift-dns/node-resolver-dwfjr"
Sep 15 22:27:10.013332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.013220    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c91b9f47-6ec9-47fe-b514-d4a26101966b-iptables-alerter-script\") pod \"iptables-alerter-xzzl6\" (UID: \"c91b9f47-6ec9-47fe-b514-d4a26101966b\") " pod="openshift-network-operator/iptables-alerter-xzzl6"
Sep 15 22:27:10.014872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.014854    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cdf53f14-911c-4dac-94d9-caf371cfc501-etc-tuned\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.015075 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.015049    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdf53f14-911c-4dac-94d9-caf371cfc501-tmp\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.023820 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.023754    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 22:27:10.023820 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.023775    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 22:27:10.023820 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.023789    2584 projected.go:196] Error preparing data for projected volume kube-api-access-rr8ds for pod openshift-network-diagnostics/network-check-target-jc5qb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:10.024041 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.023860    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds podName:752e7fb0-13a9-4aee-b255-eb0eb8c04bdf nodeName:}" failed. No retries permitted until 2026-09-15 22:27:10.523843572 +0000 UTC m=+3.142658474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8ds" (UniqueName: "kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds") pod "network-check-target-jc5qb" (UID: "752e7fb0-13a9-4aee-b255-eb0eb8c04bdf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:10.025420 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.025392    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwql5\" (UniqueName: \"kubernetes.io/projected/0b4a2eb7-70ba-4b16-8bc1-706f8f794ada-kube-api-access-nwql5\") pod \"aws-ebs-csi-driver-node-mkq2z\" (UID: \"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.025539 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.025481    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9br\" (UniqueName: \"kubernetes.io/projected/e3deae5a-e1be-4af6-8ac0-ef4976250510-kube-api-access-gr9br\") pod \"node-resolver-dwfjr\" (UID: \"e3deae5a-e1be-4af6-8ac0-ef4976250510\") " pod="openshift-dns/node-resolver-dwfjr"
Sep 15 22:27:10.026210 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.026188    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7twqw\" (UniqueName: \"kubernetes.io/projected/cdf53f14-911c-4dac-94d9-caf371cfc501-kube-api-access-7twqw\") pod \"tuned-q72l4\" (UID: \"cdf53f14-911c-4dac-94d9-caf371cfc501\") " pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.026309 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.026224    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftsqr\" (UniqueName: \"kubernetes.io/projected/c91b9f47-6ec9-47fe-b514-d4a26101966b-kube-api-access-ftsqr\") pod \"iptables-alerter-xzzl6\" (UID: \"c91b9f47-6ec9-47fe-b514-d4a26101966b\") " pod="openshift-network-operator/iptables-alerter-xzzl6"
Sep 15 22:27:10.098255 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.098223    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5nc6l"
Sep 15 22:27:10.111340 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.111313    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-llx9t"
Sep 15 22:27:10.120113 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.120092    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8jwbj"
Sep 15 22:27:10.125862 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.125844    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5fz5x"
Sep 15 22:27:10.133789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.133770    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:10.149489 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.149465    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzzl6"
Sep 15 22:27:10.157174 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.157154    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z"
Sep 15 22:27:10.163905 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.163878    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dwfjr"
Sep 15 22:27:10.170766 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.170748    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q72l4"
Sep 15 22:27:10.222093 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.222069    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 22:27:10.400100 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.400015    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 22:27:10.416527 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.416497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs\") pod \"network-metrics-daemon-pc4cx\" (UID: \"e8d2b3b9-6751-4c72-9677-dabc86c07f2a\") " pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:10.416651 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.416625    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:10.416702 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.416695    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs podName:e8d2b3b9-6751-4c72-9677-dabc86c07f2a nodeName:}" failed. No retries permitted until 2026-09-15 22:27:11.416675384 +0000 UTC m=+4.035490333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs") pod "network-metrics-daemon-pc4cx" (UID: "e8d2b3b9-6751-4c72-9677-dabc86c07f2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:10.517457 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.517427    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:10.517631 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.517597    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:10.517695 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.517671    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret podName:7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c nodeName:}" failed. No retries permitted until 2026-09-15 22:27:11.517651491 +0000 UTC m=+4.136466390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret") pod "global-pull-secret-syncer-mn2ls" (UID: "7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:10.617814 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.617775    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8ds\" (UniqueName: \"kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds\") pod \"network-check-target-jc5qb\" (UID: \"752e7fb0-13a9-4aee-b255-eb0eb8c04bdf\") " pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:10.617981 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.617935    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 22:27:10.617981 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.617953    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 22:27:10.617981 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.617963    2584 projected.go:196] Error preparing data for projected volume kube-api-access-rr8ds for pod openshift-network-diagnostics/network-check-target-jc5qb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:10.618105 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.618022    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds podName:752e7fb0-13a9-4aee-b255-eb0eb8c04bdf nodeName:}" failed. No retries permitted until 2026-09-15 22:27:11.617996365 +0000 UTC m=+4.236811254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8ds" (UniqueName: "kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds") pod "network-check-target-jc5qb" (UID: "752e7fb0-13a9-4aee-b255-eb0eb8c04bdf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:10.841836 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.841750    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 22:22:09 +0000 UTC" deadline="2028-03-07 17:03:10.761305256 +0000 UTC"
Sep 15 22:27:10.841836 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.841786    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12930h35m59.919524042s"
Sep 15 22:27:10.893200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:10.893173    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:10.893363 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:10.893298    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:11.027518 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:11.027330    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3deae5a_e1be_4af6_8ac0_ef4976250510.slice/crio-3906cc545bb50de70f20fc17440f2b0a08407cbbd11689c76fede23b4335616a WatchSource:0}: Error finding container 3906cc545bb50de70f20fc17440f2b0a08407cbbd11689c76fede23b4335616a: Status 404 returned error can't find the container with id 3906cc545bb50de70f20fc17440f2b0a08407cbbd11689c76fede23b4335616a
Sep 15 22:27:11.029057 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:11.028929    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4a2eb7_70ba_4b16_8bc1_706f8f794ada.slice/crio-1d813cb5d94627ebb964765e0d24e159653a85bad274fed02d5b26ddc4de5ead WatchSource:0}: Error finding container 1d813cb5d94627ebb964765e0d24e159653a85bad274fed02d5b26ddc4de5ead: Status 404 returned error can't find the container with id 1d813cb5d94627ebb964765e0d24e159653a85bad274fed02d5b26ddc4de5ead
Sep 15 22:27:11.035588 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:11.033831    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e67238_6768_4b3c_89f2_a17e61edee60.slice/crio-5baf6b0431fdec343d5afbeae7201c6baa6bc411ae08619f1d24980b149cd818 WatchSource:0}: Error finding container 5baf6b0431fdec343d5afbeae7201c6baa6bc411ae08619f1d24980b149cd818: Status 404 returned error can't find the container with id 5baf6b0431fdec343d5afbeae7201c6baa6bc411ae08619f1d24980b149cd818
Sep 15 22:27:11.035588 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:11.035533    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c5e1a3_16e0_4171_a9a2_df5f8eba5318.slice/crio-6e95fa6258d91dc092122586c8d71d49b5a5f2b82e91f941f4ed0e40c9771375 WatchSource:0}: Error finding container 6e95fa6258d91dc092122586c8d71d49b5a5f2b82e91f941f4ed0e40c9771375: Status 404 returned error can't find the container with id 6e95fa6258d91dc092122586c8d71d49b5a5f2b82e91f941f4ed0e40c9771375
Sep 15 22:27:11.036424 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:11.036299    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91b9f47_6ec9_47fe_b514_d4a26101966b.slice/crio-0f9eaad5527702e8bc43c0ee1885fa02ffb45dfd02463893aec1fe7f67142d69 WatchSource:0}: Error finding container 0f9eaad5527702e8bc43c0ee1885fa02ffb45dfd02463893aec1fe7f67142d69: Status 404 returned error can't find the container with id 0f9eaad5527702e8bc43c0ee1885fa02ffb45dfd02463893aec1fe7f67142d69
Sep 15 22:27:11.037339 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:11.037082    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38aa201f_97b9_443e_b246_dd19d606eb43.slice/crio-3a8d9c4867cc923e04e1a4ecd6fe1c49297ddf00e34383652ff8002d24225d2d WatchSource:0}: Error finding container 3a8d9c4867cc923e04e1a4ecd6fe1c49297ddf00e34383652ff8002d24225d2d: Status 404 returned error can't find the container with id 3a8d9c4867cc923e04e1a4ecd6fe1c49297ddf00e34383652ff8002d24225d2d
Sep 15 22:27:11.038443 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:11.038385    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc298fcdd_c5f5_4410_995a_8d66952752b8.slice/crio-b2e7de2b437155b0a29fc21d956fe365eadb64a932af0dfe039b2c1b3190ef9a WatchSource:0}: Error finding container b2e7de2b437155b0a29fc21d956fe365eadb64a932af0dfe039b2c1b3190ef9a: Status 404 returned error can't find the container with id b2e7de2b437155b0a29fc21d956fe365eadb64a932af0dfe039b2c1b3190ef9a
Sep 15 22:27:11.040053 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:11.040007    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf53f14_911c_4dac_94d9_caf371cfc501.slice/crio-c00bf3640e4852ae21e4bd383a948fa6e2b61496401542e35a0c6564e3d583a9 WatchSource:0}: Error finding container c00bf3640e4852ae21e4bd383a948fa6e2b61496401542e35a0c6564e3d583a9: Status 404 returned error can't find the container with id c00bf3640e4852ae21e4bd383a948fa6e2b61496401542e35a0c6564e3d583a9
Sep 15 22:27:11.041189 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:11.041166    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e931f99_26f8_440d_a9b9_269517778506.slice/crio-cd8750a59fa76c0f15debbe1915f5da7f72c0c86d027937e6ba58d9f13f4b898 WatchSource:0}: Error finding container cd8750a59fa76c0f15debbe1915f5da7f72c0c86d027937e6ba58d9f13f4b898: Status 404 returned error can't find the container with id cd8750a59fa76c0f15debbe1915f5da7f72c0c86d027937e6ba58d9f13f4b898
Sep 15 22:27:11.424643 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.424581    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs\") pod \"network-metrics-daemon-pc4cx\" (UID: \"e8d2b3b9-6751-4c72-9677-dabc86c07f2a\") " pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:11.424751 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:11.424678    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:11.424751 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:11.424725    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs podName:e8d2b3b9-6751-4c72-9677-dabc86c07f2a nodeName:}" failed. No retries permitted until 2026-09-15 22:27:13.424712684 +0000 UTC m=+6.043527583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs") pod "network-metrics-daemon-pc4cx" (UID: "e8d2b3b9-6751-4c72-9677-dabc86c07f2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:11.525803 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.525770    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:11.525928 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:11.525912    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:11.525985 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:11.525968    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret podName:7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c nodeName:}" failed. No retries permitted until 2026-09-15 22:27:13.525954276 +0000 UTC m=+6.144769165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret") pod "global-pull-secret-syncer-mn2ls" (UID: "7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:11.626691 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.626655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8ds\" (UniqueName: \"kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds\") pod \"network-check-target-jc5qb\" (UID: \"752e7fb0-13a9-4aee-b255-eb0eb8c04bdf\") " pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:11.626831 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:11.626817    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 22:27:11.626874 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:11.626839    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 22:27:11.626874 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:11.626849    2584 projected.go:196] Error preparing data for projected volume kube-api-access-rr8ds for pod openshift-network-diagnostics/network-check-target-jc5qb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:11.626942 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:11.626908    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds podName:752e7fb0-13a9-4aee-b255-eb0eb8c04bdf nodeName:}" failed. No retries permitted until 2026-09-15 22:27:13.626892521 +0000 UTC m=+6.245707406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8ds" (UniqueName: "kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds") pod "network-check-target-jc5qb" (UID: "752e7fb0-13a9-4aee-b255-eb0eb8c04bdf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:11.842439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.842338    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 22:22:09 +0000 UTC" deadline="2028-05-22 18:31:46.619984818 +0000 UTC"
Sep 15 22:27:11.842439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.842400    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14756h4m34.777589844s"
Sep 15 22:27:11.892830 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.892797    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:11.893019 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:11.892934    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:11.893360 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.893340    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:11.893469 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:11.893449    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:11.911444 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.911371    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzzl6" event={"ID":"c91b9f47-6ec9-47fe-b514-d4a26101966b","Type":"ContainerStarted","Data":"0f9eaad5527702e8bc43c0ee1885fa02ffb45dfd02463893aec1fe7f67142d69"}
Sep 15 22:27:11.921086 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.921019    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fz5x" event={"ID":"86c5e1a3-16e0-4171-a9a2-df5f8eba5318","Type":"ContainerStarted","Data":"6e95fa6258d91dc092122586c8d71d49b5a5f2b82e91f941f4ed0e40c9771375"}
Sep 15 22:27:11.930472 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.930376    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8jwbj" event={"ID":"c8e67238-6768-4b3c-89f2-a17e61edee60","Type":"ContainerStarted","Data":"5baf6b0431fdec343d5afbeae7201c6baa6bc411ae08619f1d24980b149cd818"}
Sep 15 22:27:11.934532 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.934447    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z" event={"ID":"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada","Type":"ContainerStarted","Data":"1d813cb5d94627ebb964765e0d24e159653a85bad274fed02d5b26ddc4de5ead"}
Sep 15 22:27:11.942464 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.942436    2584 generic.go:356] "Generic (PLEG): container finished" podID="df62c586d424be3422da26b000edf8eb" containerID="823626d100df984436abb52b96ea505ee5074901c11c3824f9bcda2aed57e1f0" exitCode=0
Sep 15 22:27:11.942564 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.942536    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal" event={"ID":"df62c586d424be3422da26b000edf8eb","Type":"ContainerDied","Data":"823626d100df984436abb52b96ea505ee5074901c11c3824f9bcda2aed57e1f0"}
Sep 15 22:27:11.950962 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.950937    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-llx9t" event={"ID":"c298fcdd-c5f5-4410-995a-8d66952752b8","Type":"ContainerStarted","Data":"b2e7de2b437155b0a29fc21d956fe365eadb64a932af0dfe039b2c1b3190ef9a"}
Sep 15 22:27:11.959702 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.959675    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5nc6l" event={"ID":"38aa201f-97b9-443e-b246-dd19d606eb43","Type":"ContainerStarted","Data":"3a8d9c4867cc923e04e1a4ecd6fe1c49297ddf00e34383652ff8002d24225d2d"}
Sep 15 22:27:11.967839 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.967815    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dwfjr" event={"ID":"e3deae5a-e1be-4af6-8ac0-ef4976250510","Type":"ContainerStarted","Data":"3906cc545bb50de70f20fc17440f2b0a08407cbbd11689c76fede23b4335616a"}
Sep 15 22:27:11.980424 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.979832    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-116.ec2.internal" event={"ID":"69eda65fcd513f716093f5dd7f2f0e50","Type":"ContainerStarted","Data":"c022fe6ef5c2f2b24c3119b35d45792ba5678c811ef938294149458780038433"}
Sep 15 22:27:11.997116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:11.997009    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn" event={"ID":"5e931f99-26f8-440d-a9b9-269517778506","Type":"ContainerStarted","Data":"cd8750a59fa76c0f15debbe1915f5da7f72c0c86d027937e6ba58d9f13f4b898"}
Sep 15 22:27:12.011826 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:12.011771    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q72l4" event={"ID":"cdf53f14-911c-4dac-94d9-caf371cfc501","Type":"ContainerStarted","Data":"c00bf3640e4852ae21e4bd383a948fa6e2b61496401542e35a0c6564e3d583a9"}
Sep 15 22:27:12.893123 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:12.892880    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:12.893614 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:12.893226    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:13.031899 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:13.031821    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal" event={"ID":"df62c586d424be3422da26b000edf8eb","Type":"ContainerStarted","Data":"6d5dd36acdff7a3a2b599184bf013a210dd0dec6ff8732a948f7600df589f225"}
Sep 15 22:27:13.047794 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:13.047723    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-116.ec2.internal" podStartSLOduration=5.047698897 podStartE2EDuration="5.047698897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:11.996507296 +0000 UTC m=+4.615322207" watchObservedRunningTime="2026-09-15 22:27:13.047698897 +0000 UTC m=+5.666513806"
Sep 15 22:27:13.445480 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:13.445372    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs\") pod \"network-metrics-daemon-pc4cx\" (UID: \"e8d2b3b9-6751-4c72-9677-dabc86c07f2a\") " pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:13.445641 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:13.445533    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:13.445641 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:13.445593    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs podName:e8d2b3b9-6751-4c72-9677-dabc86c07f2a nodeName:}" failed. No retries permitted until 2026-09-15 22:27:17.445575924 +0000 UTC m=+10.064390814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs") pod "network-metrics-daemon-pc4cx" (UID: "e8d2b3b9-6751-4c72-9677-dabc86c07f2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:13.546807 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:13.546211    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:13.546807 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:13.546384    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:13.546807 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:13.546442    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret podName:7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c nodeName:}" failed. No retries permitted until 2026-09-15 22:27:17.546423937 +0000 UTC m=+10.165238826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret") pod "global-pull-secret-syncer-mn2ls" (UID: "7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:13.647854 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:13.647196    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8ds\" (UniqueName: \"kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds\") pod \"network-check-target-jc5qb\" (UID: \"752e7fb0-13a9-4aee-b255-eb0eb8c04bdf\") " pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:13.647854 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:13.647424    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 22:27:13.647854 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:13.647446    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 22:27:13.647854 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:13.647461    2584 projected.go:196] Error preparing data for projected volume kube-api-access-rr8ds for pod openshift-network-diagnostics/network-check-target-jc5qb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:13.647854 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:13.647521    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds podName:752e7fb0-13a9-4aee-b255-eb0eb8c04bdf nodeName:}" failed. No retries permitted until 2026-09-15 22:27:17.647503364 +0000 UTC m=+10.266318257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8ds" (UniqueName: "kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds") pod "network-check-target-jc5qb" (UID: "752e7fb0-13a9-4aee-b255-eb0eb8c04bdf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:13.894872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:13.893170    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:13.894872 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:13.893307    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:13.894872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:13.893682    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:13.894872 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:13.893786    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:14.893751 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:14.893293    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:14.893751 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:14.893421    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:15.892380 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:15.892314    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:15.893151 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:15.892450    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:15.893151 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:15.892332    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:15.893151 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:15.892600    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:16.892643 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:16.892572    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:16.892996 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:16.892695    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:17.478531 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:17.478496    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs\") pod \"network-metrics-daemon-pc4cx\" (UID: \"e8d2b3b9-6751-4c72-9677-dabc86c07f2a\") " pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:17.478726 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:17.478653    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:17.478726 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:17.478720    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs podName:e8d2b3b9-6751-4c72-9677-dabc86c07f2a nodeName:}" failed. No retries permitted until 2026-09-15 22:27:25.478701009 +0000 UTC m=+18.097515909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs") pod "network-metrics-daemon-pc4cx" (UID: "e8d2b3b9-6751-4c72-9677-dabc86c07f2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:17.578972 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:17.578886    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:17.579128 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:17.579028    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:17.579128 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:17.579088    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret podName:7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c nodeName:}" failed. No retries permitted until 2026-09-15 22:27:25.579070093 +0000 UTC m=+18.197884983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret") pod "global-pull-secret-syncer-mn2ls" (UID: "7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:17.680237 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:17.680149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8ds\" (UniqueName: \"kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds\") pod \"network-check-target-jc5qb\" (UID: \"752e7fb0-13a9-4aee-b255-eb0eb8c04bdf\") " pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:17.680466 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:17.680302    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 22:27:17.680466 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:17.680325    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 22:27:17.680466 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:17.680338    2584 projected.go:196] Error preparing data for projected volume kube-api-access-rr8ds for pod openshift-network-diagnostics/network-check-target-jc5qb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:17.680466 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:17.680402    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds podName:752e7fb0-13a9-4aee-b255-eb0eb8c04bdf nodeName:}" failed. No retries permitted until 2026-09-15 22:27:25.680382393 +0000 UTC m=+18.299197294 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8ds" (UniqueName: "kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds") pod "network-check-target-jc5qb" (UID: "752e7fb0-13a9-4aee-b255-eb0eb8c04bdf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:17.896509 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:17.896434    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:17.896963 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:17.896544    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:17.896963 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:17.896628    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:17.896963 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:17.896740    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:18.892879 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:18.892752    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:18.893057 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:18.892890    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:19.893473 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:19.893440    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:19.893938 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:19.893574    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:19.893938 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:19.893632    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:19.893938 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:19.893782    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:20.892630 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:20.892600    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:20.892820 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:20.892721    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:21.892905 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:21.892812    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:21.893364 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:21.892957    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:21.893364 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:21.893023    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:21.893364 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:21.893123    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:22.893363 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:22.893324    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:22.893824 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:22.893435    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:23.892876 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:23.892840    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:23.893041 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:23.892849    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:23.893041 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:23.892969    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:23.893041 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:23.893030    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:24.892626 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:24.892594    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:24.893101 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:24.892722    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:25.536473 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:25.536440    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs\") pod \"network-metrics-daemon-pc4cx\" (UID: \"e8d2b3b9-6751-4c72-9677-dabc86c07f2a\") " pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:25.536652 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:25.536581    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:25.536652 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:25.536636    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs podName:e8d2b3b9-6751-4c72-9677-dabc86c07f2a nodeName:}" failed. No retries permitted until 2026-09-15 22:27:41.536622172 +0000 UTC m=+34.155437058 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs") pod "network-metrics-daemon-pc4cx" (UID: "e8d2b3b9-6751-4c72-9677-dabc86c07f2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:25.637356 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:25.637324    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:25.637526 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:25.637479    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:25.637578 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:25.637550    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret podName:7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c nodeName:}" failed. No retries permitted until 2026-09-15 22:27:41.637534075 +0000 UTC m=+34.256348965 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret") pod "global-pull-secret-syncer-mn2ls" (UID: "7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:25.738622 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:25.738590    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8ds\" (UniqueName: \"kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds\") pod \"network-check-target-jc5qb\" (UID: \"752e7fb0-13a9-4aee-b255-eb0eb8c04bdf\") " pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:25.738809 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:25.738780    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 22:27:25.738809 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:25.738806    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 22:27:25.738917 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:25.738816    2584 projected.go:196] Error preparing data for projected volume kube-api-access-rr8ds for pod openshift-network-diagnostics/network-check-target-jc5qb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:25.738917 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:25.738862    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds podName:752e7fb0-13a9-4aee-b255-eb0eb8c04bdf nodeName:}" failed. No retries permitted until 2026-09-15 22:27:41.738849215 +0000 UTC m=+34.357664100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8ds" (UniqueName: "kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds") pod "network-check-target-jc5qb" (UID: "752e7fb0-13a9-4aee-b255-eb0eb8c04bdf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:25.893377 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:25.893295    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:25.893377 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:25.893327    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:25.893838 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:25.893443    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:25.893838 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:25.893601    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:26.892989 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:26.892959    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:26.893163 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:26.893075    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:27.894239 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:27.894207    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:27.894689 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:27.894327    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:27.894689 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:27.894400    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:27.894689 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:27.894520    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:28.892382 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:28.892261    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:28.892470 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:28.892392    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:29.066146 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:29.066078    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-llx9t" event={"ID":"c298fcdd-c5f5-4410-995a-8d66952752b8","Type":"ContainerStarted","Data":"9f1844e44188318a25b7c61bf763774f2e0cac8f81d99507e478a2130bbf08c9"}
Sep 15 22:27:29.069963 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:29.069872    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5nc6l" event={"ID":"38aa201f-97b9-443e-b246-dd19d606eb43","Type":"ContainerStarted","Data":"cbe904288a778d355bbcd95cf4199f7f5db19db48cfc1c48969ce9c97e51d93f"}
Sep 15 22:27:29.077784 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:29.077298    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q72l4" event={"ID":"cdf53f14-911c-4dac-94d9-caf371cfc501","Type":"ContainerStarted","Data":"2d017674427a2781ff45ffe88f7c919aa5948afc3a63e01147bfce8422fcf6ee"}
Sep 15 22:27:29.086036 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:29.085952    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-116.ec2.internal" podStartSLOduration=20.085933809 podStartE2EDuration="20.085933809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:13.048538414 +0000 UTC m=+5.667353323" watchObservedRunningTime="2026-09-15 22:27:29.085933809 +0000 UTC m=+21.704748722"
Sep 15 22:27:29.116480 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:29.116193    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5nc6l" podStartSLOduration=11.605346839 podStartE2EDuration="22.1161727s" totalImagesPullingTime="10.510825861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:29.115543769 +0000 UTC m=+21.734358688" watchObservedRunningTime="2026-09-15 22:27:29.1161727 +0000 UTC m=+21.734987611"
Sep 15 22:27:29.116915 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:29.116867    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-llx9t" podStartSLOduration=3.221097322 podStartE2EDuration="21.116855273s" totalImagesPullingTime="17.895757951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:29.087087562 +0000 UTC m=+21.705902471" watchObservedRunningTime="2026-09-15 22:27:29.116855273 +0000 UTC m=+21.735670190"
Sep 15 22:27:29.162359 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:29.162304    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q72l4" podStartSLOduration=3.315282108 podStartE2EDuration="21.162285456s" totalImagesPullingTime="17.847003348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:29.162098445 +0000 UTC m=+21.780913353" watchObservedRunningTime="2026-09-15 22:27:29.162285456 +0000 UTC m=+21.781100366"
Sep 15 22:27:29.893390 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:29.893362    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:29.893589 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:29.893362    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:29.893589 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:29.893471    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:29.893708 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:29.893604    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:30.082870 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.082699    2584 generic.go:356] "Generic (PLEG): container finished" podID="86c5e1a3-16e0-4171-a9a2-df5f8eba5318" containerID="075a2217b016f94b29e2c34bd72188874bde7f02cb7f8cc81e507a394d79318e" exitCode=0
Sep 15 22:27:30.083772 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.082770    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fz5x" event={"ID":"86c5e1a3-16e0-4171-a9a2-df5f8eba5318","Type":"ContainerDied","Data":"075a2217b016f94b29e2c34bd72188874bde7f02cb7f8cc81e507a394d79318e"}
Sep 15 22:27:30.084610 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.084467    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8jwbj" event={"ID":"c8e67238-6768-4b3c-89f2-a17e61edee60","Type":"ContainerStarted","Data":"803068edcd22fbd5ca7b0fc68df968f8cd691509b11d0ec9a9934546571b7a22"}
Sep 15 22:27:30.086088 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.086002    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z" event={"ID":"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada","Type":"ContainerStarted","Data":"edc1537f10b79acffe892e4e5e2e24ce57944dc95b4fcd5c8dede0011ca54a56"}
Sep 15 22:27:30.087494 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.087475    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dwfjr" event={"ID":"e3deae5a-e1be-4af6-8ac0-ef4976250510","Type":"ContainerStarted","Data":"cdf80d59307aca8a181f42aa7b09dee1a74eade6e479b218545a1fe4ea0e891f"}
Sep 15 22:27:30.090576 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.090559    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn" event={"ID":"5e931f99-26f8-440d-a9b9-269517778506","Type":"ContainerStarted","Data":"11efd8275a8b746d5c9bb578a8bfa42d25e929c36541521a1de2962af0e5b7e0"}
Sep 15 22:27:30.090674 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.090629    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn" event={"ID":"5e931f99-26f8-440d-a9b9-269517778506","Type":"ContainerStarted","Data":"264b5c7faaacde0507eabdf33b7594d6590773e168265773ee71ec6ee2e204a2"}
Sep 15 22:27:30.090734 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.090692    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn" event={"ID":"5e931f99-26f8-440d-a9b9-269517778506","Type":"ContainerStarted","Data":"433a0c57815bb6a0a8d7c1a99b2cd13d76225da2b4fce758f7aa790d9d5f303b"}
Sep 15 22:27:30.090792 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.090744    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn" event={"ID":"5e931f99-26f8-440d-a9b9-269517778506","Type":"ContainerStarted","Data":"65b4b934b1dbf6c988148ded0dc423e6541d439e9bdf57a4aeb5990e92a9b0b2"}
Sep 15 22:27:30.090845 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.090797    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn" event={"ID":"5e931f99-26f8-440d-a9b9-269517778506","Type":"ContainerStarted","Data":"c3e9831ec6e84ec1b1d9e6ab2e862af6ebe4d0fa556a13474381cf327c2b7a8a"}
Sep 15 22:27:30.090900 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.090846    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn" event={"ID":"5e931f99-26f8-440d-a9b9-269517778506","Type":"ContainerStarted","Data":"8574a8383c7bc52393b00af98f34a2e3b007e225cc04773c4af4cffd8924501e"}
Sep 15 22:27:30.118467 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.118427    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8jwbj" podStartSLOduration=4.268511618 podStartE2EDuration="22.118414868s" totalImagesPullingTime="17.84990325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:30.118227175 +0000 UTC m=+22.737042084" watchObservedRunningTime="2026-09-15 22:27:30.118414868 +0000 UTC m=+22.737229775"
Sep 15 22:27:30.132573 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.132539    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dwfjr" podStartSLOduration=4.275344798 podStartE2EDuration="22.132528098s" totalImagesPullingTime="17.8571833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:30.132345863 +0000 UTC m=+22.751160770" watchObservedRunningTime="2026-09-15 22:27:30.132528098 +0000 UTC m=+22.751343005"
Sep 15 22:27:30.375374 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.375351    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 22:27:30.851103 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.850972    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T22:27:30.375369329Z","UUID":"d9deba9f-67ec-4283-8e7e-580b1452e8df","Handler":null,"Name":"","Endpoint":""}
Sep 15 22:27:30.852551 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.852525    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 22:27:30.852675 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.852577    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 22:27:30.893256 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:30.893228    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:30.893385 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:30.893328    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:31.096908 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:31.096860    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z" event={"ID":"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada","Type":"ContainerStarted","Data":"668c1b1a7ab4e276e859e08912841a9176964f7889ebe6c707c9ab74d7456453"}
Sep 15 22:27:31.099908 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:31.099879    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzzl6" event={"ID":"c91b9f47-6ec9-47fe-b514-d4a26101966b","Type":"ContainerStarted","Data":"4cc2d77fb26eec0ba2adfa7fc9b217e93eced76bcc53dc1d981773cab50308a7"}
Sep 15 22:27:31.118563 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:31.118484    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xzzl6" podStartSLOduration=5.269726153 podStartE2EDuration="23.11846767s" totalImagesPullingTime="17.848741517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:31.1183431 +0000 UTC m=+23.737158008" watchObservedRunningTime="2026-09-15 22:27:31.11846767 +0000 UTC m=+23.737282579"
Sep 15 22:27:31.892681 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:31.892596    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:31.892681 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:31.892622    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:31.892891 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:31.892719    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:31.892891 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:31.892857    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:32.103354 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:32.103320    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z" event={"ID":"0b4a2eb7-70ba-4b16-8bc1-706f8f794ada","Type":"ContainerStarted","Data":"661431fc17c9a5fbadacc90b8eff596a2ee0f34f9964917f30decfbfe344ed9e"}
Sep 15 22:27:32.106439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:32.106415    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn" event={"ID":"5e931f99-26f8-440d-a9b9-269517778506","Type":"ContainerStarted","Data":"99d1be1fce522d6b1a646bd2f5c46437e6dd382f59e67af4ced843e8dbd1d4e7"}
Sep 15 22:27:32.126759 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:32.126697    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkq2z" podStartSLOduration=4.063135342 podStartE2EDuration="24.126678036s" totalImagesPullingTime="20.063542694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:32.12649702 +0000 UTC m=+24.745311930" watchObservedRunningTime="2026-09-15 22:27:32.126678036 +0000 UTC m=+24.745492960"
Sep 15 22:27:32.713544 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:32.713507    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5nc6l"
Sep 15 22:27:32.714532 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:32.714515    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5nc6l"
Sep 15 22:27:32.892467 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:32.892441    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:32.892611 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:32.892532    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:33.893235 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:33.893035    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:33.893655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:33.893052    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:33.893655 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:33.893384    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:33.893655 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:33.893517    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:34.114700 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:34.114624    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn" event={"ID":"5e931f99-26f8-440d-a9b9-269517778506","Type":"ContainerStarted","Data":"96e170918a7ba25e0a6684eb6d853d8fbde47c95f86c40015cfbf0831d53c5d6"}
Sep 15 22:27:34.114964 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:34.114944    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:34.130683 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:34.130658    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:34.144050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:34.143997    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn" podStartSLOduration=8.244925864 podStartE2EDuration="26.143984444s" totalImagesPullingTime="17.89905858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:34.14338095 +0000 UTC m=+26.762195859" watchObservedRunningTime="2026-09-15 22:27:34.143984444 +0000 UTC m=+26.762799352"
Sep 15 22:27:34.892827 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:34.892798    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:34.893011 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:34.892905    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:35.118434 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:35.118401    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:35.119224 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:35.118493    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:35.134449 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:35.134425    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:27:35.708646 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:35.708610    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mn2ls"]
Sep 15 22:27:35.708813 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:35.708712    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:35.708893 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:35.708825    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:35.712513 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:35.712487    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jc5qb"]
Sep 15 22:27:35.712681 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:35.712563    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:35.713053 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:35.712994    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:35.717083 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:35.717062    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pc4cx"]
Sep 15 22:27:35.717203 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:35.717187    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:35.717326 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:35.717306    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:36.121123 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:36.121090    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fz5x" event={"ID":"86c5e1a3-16e0-4171-a9a2-df5f8eba5318","Type":"ContainerStarted","Data":"fa65bfc863e779ed3ac5b6884f4f93ea1c4d018a3eda102406b2d4dd61316ee4"}
Sep 15 22:27:37.125103 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:37.124926    2584 generic.go:356] "Generic (PLEG): container finished" podID="86c5e1a3-16e0-4171-a9a2-df5f8eba5318" containerID="fa65bfc863e779ed3ac5b6884f4f93ea1c4d018a3eda102406b2d4dd61316ee4" exitCode=0
Sep 15 22:27:37.125479 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:37.124998    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fz5x" event={"ID":"86c5e1a3-16e0-4171-a9a2-df5f8eba5318","Type":"ContainerDied","Data":"fa65bfc863e779ed3ac5b6884f4f93ea1c4d018a3eda102406b2d4dd61316ee4"}
Sep 15 22:27:37.893336 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:37.893307    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:37.893492 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:37.893343    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:37.893492 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:37.893405    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:37.893492 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:37.893477    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:37.893659 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:37.893471    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:37.893659 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:37.893567    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:39.131842 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:39.131806    2584 generic.go:356] "Generic (PLEG): container finished" podID="86c5e1a3-16e0-4171-a9a2-df5f8eba5318" containerID="2eb36474a6ac95429794dec3b075ccc99289b15c2bf2dfd8b396a5bd5d7a2b7e" exitCode=0
Sep 15 22:27:39.132260 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:39.131861    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fz5x" event={"ID":"86c5e1a3-16e0-4171-a9a2-df5f8eba5318","Type":"ContainerDied","Data":"2eb36474a6ac95429794dec3b075ccc99289b15c2bf2dfd8b396a5bd5d7a2b7e"}
Sep 15 22:27:39.596068 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:39.595996    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5nc6l"
Sep 15 22:27:39.596228 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:39.596102    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 22:27:39.596623 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:39.596601    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5nc6l"
Sep 15 22:27:39.893370 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:39.893275    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:39.893512 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:39.893281    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:39.893512 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:39.893406    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:39.893512 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:39.893419    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:39.893690 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:39.893513    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:39.893690 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:39.893614    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:41.138328 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:41.138299    2584 generic.go:356] "Generic (PLEG): container finished" podID="86c5e1a3-16e0-4171-a9a2-df5f8eba5318" containerID="8f32b4cad68c4ec169ca3edca317a879e05596ed06083074737ef8bc76aa4488" exitCode=0
Sep 15 22:27:41.138702 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:41.138343    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fz5x" event={"ID":"86c5e1a3-16e0-4171-a9a2-df5f8eba5318","Type":"ContainerDied","Data":"8f32b4cad68c4ec169ca3edca317a879e05596ed06083074737ef8bc76aa4488"}
Sep 15 22:27:41.561471 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:41.561445    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs\") pod \"network-metrics-daemon-pc4cx\" (UID: \"e8d2b3b9-6751-4c72-9677-dabc86c07f2a\") " pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:41.561612 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:41.561586    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:41.561652 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:41.561645    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs podName:e8d2b3b9-6751-4c72-9677-dabc86c07f2a nodeName:}" failed. No retries permitted until 2026-09-15 22:28:13.561629373 +0000 UTC m=+66.180444259 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs") pod "network-metrics-daemon-pc4cx" (UID: "e8d2b3b9-6751-4c72-9677-dabc86c07f2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 22:27:41.662280 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:41.662237    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:41.662389 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:41.662374    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:41.662444 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:41.662430    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret podName:7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c nodeName:}" failed. No retries permitted until 2026-09-15 22:28:13.662413809 +0000 UTC m=+66.281228697 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret") pod "global-pull-secret-syncer-mn2ls" (UID: "7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c") : object "kube-system"/"original-pull-secret" not registered
Sep 15 22:27:41.762734 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:41.762704    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8ds\" (UniqueName: \"kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds\") pod \"network-check-target-jc5qb\" (UID: \"752e7fb0-13a9-4aee-b255-eb0eb8c04bdf\") " pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:41.762870 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:41.762856    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 22:27:41.762927 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:41.762881    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 22:27:41.762927 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:41.762895    2584 projected.go:196] Error preparing data for projected volume kube-api-access-rr8ds for pod openshift-network-diagnostics/network-check-target-jc5qb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:41.763003 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:41.762954    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds podName:752e7fb0-13a9-4aee-b255-eb0eb8c04bdf nodeName:}" failed. No retries permitted until 2026-09-15 22:28:13.762937499 +0000 UTC m=+66.381752390 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rr8ds" (UniqueName: "kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds") pod "network-check-target-jc5qb" (UID: "752e7fb0-13a9-4aee-b255-eb0eb8c04bdf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 22:27:41.893235 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:41.893176    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:41.893466 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:41.893176    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:41.893466 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:41.893329    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:41.893466 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:41.893323    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pc4cx" podUID="e8d2b3b9-6751-4c72-9677-dabc86c07f2a"
Sep 15 22:27:41.893466 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:41.893436    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jc5qb" podUID="752e7fb0-13a9-4aee-b255-eb0eb8c04bdf"
Sep 15 22:27:41.893657 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:41.893518    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mn2ls" podUID="7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c"
Sep 15 22:27:42.710817 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.710740    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-116.ec2.internal" event="NodeReady"
Sep 15 22:27:42.711253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.710899    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 22:27:42.753975 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.753937    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"]
Sep 15 22:27:42.755843 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.755809    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-687df9c7d4-t72hr"]
Sep 15 22:27:42.755963 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.755931    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:42.757749 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.757729    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.759473 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.759428    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 15 22:27:42.759905 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.759776    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6df89d9594-wrhv2"]
Sep 15 22:27:42.760850 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.760394    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 22:27:42.760850 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.760575    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 22:27:42.761797 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.761777    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"]
Sep 15 22:27:42.761903 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.761887    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.763538 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.763515    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"]
Sep 15 22:27:42.763642 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.763609    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"
Sep 15 22:27:42.765714 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.765696    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 22:27:42.766073 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.766057    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:27:42.775318 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.775295    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 22:27:42.775776 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.775757    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5kgc5"]
Sep 15 22:27:42.777695 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.777676    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5kgc5"
Sep 15 22:27:42.777942 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.777893    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 22:27:42.779232 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.779210    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-28xs6\""
Sep 15 22:27:42.779609 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.779590    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6tvss\""
Sep 15 22:27:42.780595 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.780577    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"]
Sep 15 22:27:42.780981 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.780964    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 15 22:27:42.781151 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.781138    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 15 22:27:42.781439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.781322    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 15 22:27:42.781439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.781407    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 22:27:42.781574 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.781547    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 15 22:27:42.781676 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.781628    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 22:27:42.782197 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.782172    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 15 22:27:42.785589 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.782455    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 15 22:27:42.785589 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.782729    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 15 22:27:42.785589 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.781628    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 15 22:27:42.785589 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.784439    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"
Sep 15 22:27:42.785589 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.784931    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-9xx4f"]
Sep 15 22:27:42.785589 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.785146    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 15 22:27:42.785943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.785653    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dzgwm\""
Sep 15 22:27:42.786145 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.786050    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-s9xcw\""
Sep 15 22:27:42.790446 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.790181    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6vmjf\""
Sep 15 22:27:42.790446 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.790375    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 15 22:27:42.790581 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.787716    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 15 22:27:42.792357 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.791003    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 15 22:27:42.792357 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.791324    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mwpvf\""
Sep 15 22:27:42.792357 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.791888    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 15 22:27:42.793416 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.792929    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 15 22:27:42.794661 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.794640    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-9x825"]
Sep 15 22:27:42.795555 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.795535    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:42.797000 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.796979    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"]
Sep 15 22:27:42.797102 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.797084    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:42.797229 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.797209    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5kgc5"]
Sep 15 22:27:42.797356 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.797246    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-687df9c7d4-t72hr"]
Sep 15 22:27:42.797413 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.797365    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 22:27:42.797474 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.797417    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"]
Sep 15 22:27:42.798104 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.798082    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 15 22:27:42.798281 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.798242    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 15 22:27:42.798379 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.798310    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 15 22:27:42.798514 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.798498    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 15 22:27:42.798665 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.798642    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-79sgv\""
Sep 15 22:27:42.799316 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.799256    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sm96j"]
Sep 15 22:27:42.801386 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.801369    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-h6msr"]
Sep 15 22:27:42.801562 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.801544    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:42.804811 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.804794    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 15 22:27:42.805156 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.805138    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9x825"]
Sep 15 22:27:42.805872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.805842    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"]
Sep 15 22:27:42.805997 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.805980    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"]
Sep 15 22:27:42.806088 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.805550    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 15 22:27:42.806088 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.805660    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-h6msr"
Sep 15 22:27:42.806193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.805786    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 22:27:42.806193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.805347    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 15 22:27:42.806323 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.805475    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 15 22:27:42.806613 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.806586    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 15 22:27:42.806692 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.806593    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vpjst\""
Sep 15 22:27:42.807339 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.807310    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8pmw9\""
Sep 15 22:27:42.807417 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.807391    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 15 22:27:42.807523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.807506    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 22:27:42.807605 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.807507    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 22:27:42.807669 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.807619    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 22:27:42.808055 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.808035    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6hpf8\""
Sep 15 22:27:42.812504 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.812349    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 22:27:42.813142 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.813070    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6df89d9594-wrhv2"]
Sep 15 22:27:42.813142 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.813103    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 22:27:42.813433 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.813413    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hbzz8\""
Sep 15 22:27:42.813583 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.813562    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-9xx4f"]
Sep 15 22:27:42.815848 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.815400    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 15 22:27:42.820529 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.820507    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 15 22:27:42.821798 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.821775    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-h6msr"]
Sep 15 22:27:42.825545 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.825517    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sm96j"]
Sep 15 22:27:42.857774 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.857744    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:42.863451 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.863431    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bw662"]
Sep 15 22:27:42.866620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.866603    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bw662"
Sep 15 22:27:42.869502 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869475    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxrjt\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-kube-api-access-qxrjt\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.869587 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869515    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/702fee44-5a8b-4364-a00f-957fab8661dd-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qc5w9\" (UID: \"702fee44-5a8b-4364-a00f-957fab8661dd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"
Sep 15 22:27:42.869587 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869542    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/67b271e9-a2b8-4f33-be0b-102e434588ac-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:42.869587 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869579    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.869755 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869607    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.869755 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869640    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-trusted-ca\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.869755 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869665    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-bound-sa-token\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.869755 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869682    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-96r9x\" (UID: \"7f812033-940b-4211-bd1c-2eceec145d80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:27:42.869755 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869705    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-certificates\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.869985 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869767    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8nf4\" (UniqueName: \"kubernetes.io/projected/7f6ab2a4-b56d-45b9-81d8-1319abade096-kube-api-access-k8nf4\") pod \"volume-data-source-validator-86695b7cc7-5kgc5\" (UID: \"7f6ab2a4-b56d-45b9-81d8-1319abade096\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5kgc5"
Sep 15 22:27:42.869985 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869794    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpc2l\" (UniqueName: \"kubernetes.io/projected/67b271e9-a2b8-4f33-be0b-102e434588ac-kube-api-access-gpc2l\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:42.869985 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869837    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.869985 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869915    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-ca-trust-extracted\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.869985 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869950    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/307013b9-cf57-4e9e-bddd-e9d51ef7c57b-serving-cert\") pod \"service-ca-operator-cf7c776b7-c4qxc\" (UID: \"307013b9-cf57-4e9e-bddd-e9d51ef7c57b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"
Sep 15 22:27:42.869985 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869974    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/307013b9-cf57-4e9e-bddd-e9d51ef7c57b-config\") pod \"service-ca-operator-cf7c776b7-c4qxc\" (UID: \"307013b9-cf57-4e9e-bddd-e9d51ef7c57b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"
Sep 15 22:27:42.870233 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.869996    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-installation-pull-secrets\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.870233 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.870011    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf795\" (UniqueName: \"kubernetes.io/projected/7f812033-940b-4211-bd1c-2eceec145d80-kube-api-access-xf795\") pod \"cluster-samples-operator-86499c89c4-96r9x\" (UID: \"7f812033-940b-4211-bd1c-2eceec145d80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:27:42.870233 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.870101    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdvr\" (UniqueName: \"kubernetes.io/projected/307013b9-cf57-4e9e-bddd-e9d51ef7c57b-kube-api-access-nsdvr\") pod \"service-ca-operator-cf7c776b7-c4qxc\" (UID: \"307013b9-cf57-4e9e-bddd-e9d51ef7c57b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"
Sep 15 22:27:42.870233 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.870140    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/702fee44-5a8b-4364-a00f-957fab8661dd-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qc5w9\" (UID: \"702fee44-5a8b-4364-a00f-957fab8661dd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"
Sep 15 22:27:42.870233 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.870164    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-default-certificate\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.870233 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.870210    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-stats-auth\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.870531 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.870234    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m5v7\" (UniqueName: \"kubernetes.io/projected/f53b8e84-9062-4265-9b5e-4d4b40ad7020-kube-api-access-2m5v7\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.870531 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.870244    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 22:27:42.870531 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.870297    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-image-registry-private-configuration\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.870531 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.870323    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfw72\" (UniqueName: \"kubernetes.io/projected/702fee44-5a8b-4364-a00f-957fab8661dd-kube-api-access-zfw72\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qc5w9\" (UID: \"702fee44-5a8b-4364-a00f-957fab8661dd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"
Sep 15 22:27:42.870531 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.870360    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:42.870531 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.870504    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 22:27:42.870884 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.870867    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vrlmr\""
Sep 15 22:27:42.870954 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.870945    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 22:27:42.899446 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.899422    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bw662"]
Sep 15 22:27:42.971374 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971305    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc2ba350-9c05-4b47-9f16-8b19093b9137-service-ca-bundle\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:42.971374 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971351    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-certificates\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.971554 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971393    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b68f8430-b1b6-46d9-8358-74f04b66c9d6-config\") pod \"console-operator-696884fb9c-9xx4f\" (UID: \"b68f8430-b1b6-46d9-8358-74f04b66c9d6\") " pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:42.971554 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971417    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njk87\" (UniqueName: \"kubernetes.io/projected/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-kube-api-access-njk87\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:42.971554 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971454    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-stats-auth\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.971554 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971510    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc2ba350-9c05-4b47-9f16-8b19093b9137-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:42.971747 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971560    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc2ba350-9c05-4b47-9f16-8b19093b9137-tmp\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:42.971747 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971593    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-installation-pull-secrets\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.971747 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971622    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/702fee44-5a8b-4364-a00f-957fab8661dd-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qc5w9\" (UID: \"702fee44-5a8b-4364-a00f-957fab8661dd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"
Sep 15 22:27:42.971747 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971649    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-default-certificate\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.971747 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971674    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdvr\" (UniqueName: \"kubernetes.io/projected/307013b9-cf57-4e9e-bddd-e9d51ef7c57b-kube-api-access-nsdvr\") pod \"service-ca-operator-cf7c776b7-c4qxc\" (UID: \"307013b9-cf57-4e9e-bddd-e9d51ef7c57b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"
Sep 15 22:27:42.971747 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971704    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-image-registry-private-configuration\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.971747 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971729    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5drg\" (UniqueName: \"kubernetes.io/projected/4c4b4d38-64d6-408a-bf2f-591822b41ac2-kube-api-access-s5drg\") pod \"ingress-canary-bw662\" (UID: \"4c4b4d38-64d6-408a-bf2f-591822b41ac2\") " pod="openshift-ingress-canary/ingress-canary-bw662"
Sep 15 22:27:42.972080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971753    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:42.972080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971778    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b68f8430-b1b6-46d9-8358-74f04b66c9d6-serving-cert\") pod \"console-operator-696884fb9c-9xx4f\" (UID: \"b68f8430-b1b6-46d9-8358-74f04b66c9d6\") " pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:42.972080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971811    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwtzn\" (UniqueName: \"kubernetes.io/projected/b68f8430-b1b6-46d9-8358-74f04b66c9d6-kube-api-access-gwtzn\") pod \"console-operator-696884fb9c-9xx4f\" (UID: \"b68f8430-b1b6-46d9-8358-74f04b66c9d6\") " pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:42.972080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971838    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/702fee44-5a8b-4364-a00f-957fab8661dd-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qc5w9\" (UID: \"702fee44-5a8b-4364-a00f-957fab8661dd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"
Sep 15 22:27:42.972080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971861    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fc2ba350-9c05-4b47-9f16-8b19093b9137-snapshots\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:42.972080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971892    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.972080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971948    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b68f8430-b1b6-46d9-8358-74f04b66c9d6-trusted-ca\") pod \"console-operator-696884fb9c-9xx4f\" (UID: \"b68f8430-b1b6-46d9-8358-74f04b66c9d6\") " pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:42.972080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971977    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-certificates\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.972080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.971991    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c4b4d38-64d6-408a-bf2f-591822b41ac2-cert\") pod \"ingress-canary-bw662\" (UID: \"4c4b4d38-64d6-408a-bf2f-591822b41ac2\") " pod="openshift-ingress-canary/ingress-canary-bw662"
Sep 15 22:27:42.972080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.972018    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-trusted-ca\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.972080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.972048    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2ba350-9c05-4b47-9f16-8b19093b9137-serving-cert\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:42.972080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.972079    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8nf4\" (UniqueName: \"kubernetes.io/projected/7f6ab2a4-b56d-45b9-81d8-1319abade096-kube-api-access-k8nf4\") pod \"volume-data-source-validator-86695b7cc7-5kgc5\" (UID: \"7f6ab2a4-b56d-45b9-81d8-1319abade096\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5kgc5"
Sep 15 22:27:42.972658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.972107    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.972658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.972133    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpc2l\" (UniqueName: \"kubernetes.io/projected/67b271e9-a2b8-4f33-be0b-102e434588ac-kube-api-access-gpc2l\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:42.972658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.972164    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-ca-trust-extracted\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.972658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.972179    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/702fee44-5a8b-4364-a00f-957fab8661dd-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qc5w9\" (UID: \"702fee44-5a8b-4364-a00f-957fab8661dd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"
Sep 15 22:27:42.972658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.972190    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/307013b9-cf57-4e9e-bddd-e9d51ef7c57b-config\") pod \"service-ca-operator-cf7c776b7-c4qxc\" (UID: \"307013b9-cf57-4e9e-bddd-e9d51ef7c57b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"
Sep 15 22:27:42.972658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.972217    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m5v7\" (UniqueName: \"kubernetes.io/projected/f53b8e84-9062-4265-9b5e-4d4b40ad7020-kube-api-access-2m5v7\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.972658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.972585    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-ca-trust-extracted\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.973001 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:42.972622    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 22:27:42.973001 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.972820    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/307013b9-cf57-4e9e-bddd-e9d51ef7c57b-config\") pod \"service-ca-operator-cf7c776b7-c4qxc\" (UID: \"307013b9-cf57-4e9e-bddd-e9d51ef7c57b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"
Sep 15 22:27:42.973001 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:42.972887    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs podName:f53b8e84-9062-4265-9b5e-4d4b40ad7020 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:43.472869247 +0000 UTC m=+36.091684146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs") pod "router-default-6df89d9594-wrhv2" (UID: "f53b8e84-9062-4265-9b5e-4d4b40ad7020") : secret "router-metrics-certs-default" not found
Sep 15 22:27:42.973001 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.972989    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/307013b9-cf57-4e9e-bddd-e9d51ef7c57b-serving-cert\") pod \"service-ca-operator-cf7c776b7-c4qxc\" (UID: \"307013b9-cf57-4e9e-bddd-e9d51ef7c57b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"
Sep 15 22:27:42.973209 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.973022    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfw72\" (UniqueName: \"kubernetes.io/projected/702fee44-5a8b-4364-a00f-957fab8661dd-kube-api-access-zfw72\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qc5w9\" (UID: \"702fee44-5a8b-4364-a00f-957fab8661dd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"
Sep 15 22:27:42.973209 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.973088    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:42.973209 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.973125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-tmp-dir\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:42.973433 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:42.973414    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 22:27:42.973496 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.973464    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf795\" (UniqueName: \"kubernetes.io/projected/7f812033-940b-4211-bd1c-2eceec145d80-kube-api-access-xf795\") pod \"cluster-samples-operator-86499c89c4-96r9x\" (UID: \"7f812033-940b-4211-bd1c-2eceec145d80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:27:42.973564 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.973544    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-trusted-ca\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.973634 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:42.973622    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls podName:67b271e9-a2b8-4f33-be0b-102e434588ac nodeName:}" failed. No retries permitted until 2026-09-15 22:27:43.473605079 +0000 UTC m=+36.092419982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-gx49h" (UID: "67b271e9-a2b8-4f33-be0b-102e434588ac") : secret "cluster-monitoring-operator-tls" not found
Sep 15 22:27:42.973691 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.973656    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzm46\" (UniqueName: \"kubernetes.io/projected/2558ab4c-e858-4efa-a076-d49c1dcf1dd0-kube-api-access-fzm46\") pod \"network-check-source-6f8b768455-h6msr\" (UID: \"2558ab4c-e858-4efa-a076-d49c1dcf1dd0\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-h6msr"
Sep 15 22:27:42.973741 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.973723    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9r6z\" (UniqueName: \"kubernetes.io/projected/fc2ba350-9c05-4b47-9f16-8b19093b9137-kube-api-access-s9r6z\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:42.973804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.973787    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxrjt\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-kube-api-access-qxrjt\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.973855 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:42.973833    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle podName:f53b8e84-9062-4265-9b5e-4d4b40ad7020 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:43.47381963 +0000 UTC m=+36.092634538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle") pod "router-default-6df89d9594-wrhv2" (UID: "f53b8e84-9062-4265-9b5e-4d4b40ad7020") : configmap references non-existent config key: service-ca.crt
Sep 15 22:27:42.973914 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.973891    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/67b271e9-a2b8-4f33-be0b-102e434588ac-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:42.974020 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.973996    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.974086 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.974034    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-96r9x\" (UID: \"7f812033-940b-4211-bd1c-2eceec145d80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:27:42.974086 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.974076    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-config-volume\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:42.974190 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.974106    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-bound-sa-token\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.974811 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:42.974397    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 22:27:42.974811 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:42.974455    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls podName:7f812033-940b-4211-bd1c-2eceec145d80 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:43.474440097 +0000 UTC m=+36.093254982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-96r9x" (UID: "7f812033-940b-4211-bd1c-2eceec145d80") : secret "samples-operator-tls" not found
Sep 15 22:27:42.974811 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:42.974524    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 22:27:42.974811 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:42.974535    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687df9c7d4-t72hr: secret "image-registry-tls" not found
Sep 15 22:27:42.974811 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:42.974575    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls podName:0fc17f4f-84da-4a36-a3bc-c874fb9296e8 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:43.474562594 +0000 UTC m=+36.093377480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls") pod "image-registry-687df9c7d4-t72hr" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8") : secret "image-registry-tls" not found
Sep 15 22:27:42.974811 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.974628    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/67b271e9-a2b8-4f33-be0b-102e434588ac-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:42.976931 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.976889    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/307013b9-cf57-4e9e-bddd-e9d51ef7c57b-serving-cert\") pod \"service-ca-operator-cf7c776b7-c4qxc\" (UID: \"307013b9-cf57-4e9e-bddd-e9d51ef7c57b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"
Sep 15 22:27:42.976931 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.976899    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/702fee44-5a8b-4364-a00f-957fab8661dd-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qc5w9\" (UID: \"702fee44-5a8b-4364-a00f-957fab8661dd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"
Sep 15 22:27:42.977185 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.977163    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-installation-pull-secrets\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.977294 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.977220    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-stats-auth\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.977294 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.977248    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-default-certificate\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.977614 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.977591    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-image-registry-private-configuration\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.985571 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.985548    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-bound-sa-token\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.985669 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.985647    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8nf4\" (UniqueName: \"kubernetes.io/projected/7f6ab2a4-b56d-45b9-81d8-1319abade096-kube-api-access-k8nf4\") pod \"volume-data-source-validator-86695b7cc7-5kgc5\" (UID: \"7f6ab2a4-b56d-45b9-81d8-1319abade096\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5kgc5"
Sep 15 22:27:42.987700 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.987493    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpc2l\" (UniqueName: \"kubernetes.io/projected/67b271e9-a2b8-4f33-be0b-102e434588ac-kube-api-access-gpc2l\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:42.987700 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.987619    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf795\" (UniqueName: \"kubernetes.io/projected/7f812033-940b-4211-bd1c-2eceec145d80-kube-api-access-xf795\") pod \"cluster-samples-operator-86499c89c4-96r9x\" (UID: \"7f812033-940b-4211-bd1c-2eceec145d80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:27:42.988102 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.988078    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxrjt\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-kube-api-access-qxrjt\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:42.988191 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.988177    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m5v7\" (UniqueName: \"kubernetes.io/projected/f53b8e84-9062-4265-9b5e-4d4b40ad7020-kube-api-access-2m5v7\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:42.988395 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.988362    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdvr\" (UniqueName: \"kubernetes.io/projected/307013b9-cf57-4e9e-bddd-e9d51ef7c57b-kube-api-access-nsdvr\") pod \"service-ca-operator-cf7c776b7-c4qxc\" (UID: \"307013b9-cf57-4e9e-bddd-e9d51ef7c57b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"
Sep 15 22:27:42.990200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:42.990178    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfw72\" (UniqueName: \"kubernetes.io/projected/702fee44-5a8b-4364-a00f-957fab8661dd-kube-api-access-zfw72\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qc5w9\" (UID: \"702fee44-5a8b-4364-a00f-957fab8661dd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"
Sep 15 22:27:43.009480 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.009455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:43.074618 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.074582    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-tmp-dir\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:43.074789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.074629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzm46\" (UniqueName: \"kubernetes.io/projected/2558ab4c-e858-4efa-a076-d49c1dcf1dd0-kube-api-access-fzm46\") pod \"network-check-source-6f8b768455-h6msr\" (UID: \"2558ab4c-e858-4efa-a076-d49c1dcf1dd0\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-h6msr"
Sep 15 22:27:43.074789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.074660    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9r6z\" (UniqueName: \"kubernetes.io/projected/fc2ba350-9c05-4b47-9f16-8b19093b9137-kube-api-access-s9r6z\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:43.074789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.074731    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-config-volume\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:43.074789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.074765    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc2ba350-9c05-4b47-9f16-8b19093b9137-service-ca-bundle\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:43.075004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.074802    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b68f8430-b1b6-46d9-8358-74f04b66c9d6-config\") pod \"console-operator-696884fb9c-9xx4f\" (UID: \"b68f8430-b1b6-46d9-8358-74f04b66c9d6\") " pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:43.075004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.074826    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njk87\" (UniqueName: \"kubernetes.io/projected/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-kube-api-access-njk87\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:43.075004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.074865    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc2ba350-9c05-4b47-9f16-8b19093b9137-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:43.075004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.074899    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc2ba350-9c05-4b47-9f16-8b19093b9137-tmp\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:43.075004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.074934    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-tmp-dir\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:43.075004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.074935    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5drg\" (UniqueName: \"kubernetes.io/projected/4c4b4d38-64d6-408a-bf2f-591822b41ac2-kube-api-access-s5drg\") pod \"ingress-canary-bw662\" (UID: \"4c4b4d38-64d6-408a-bf2f-591822b41ac2\") " pod="openshift-ingress-canary/ingress-canary-bw662"
Sep 15 22:27:43.075004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.074993    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:43.075361 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.075023    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b68f8430-b1b6-46d9-8358-74f04b66c9d6-serving-cert\") pod \"console-operator-696884fb9c-9xx4f\" (UID: \"b68f8430-b1b6-46d9-8358-74f04b66c9d6\") " pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:43.075361 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.075049    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwtzn\" (UniqueName: \"kubernetes.io/projected/b68f8430-b1b6-46d9-8358-74f04b66c9d6-kube-api-access-gwtzn\") pod \"console-operator-696884fb9c-9xx4f\" (UID: \"b68f8430-b1b6-46d9-8358-74f04b66c9d6\") " pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:43.075361 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.075079    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fc2ba350-9c05-4b47-9f16-8b19093b9137-snapshots\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:43.075361 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.075112    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 22:27:43.075361 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.075119    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b68f8430-b1b6-46d9-8358-74f04b66c9d6-trusted-ca\") pod \"console-operator-696884fb9c-9xx4f\" (UID: \"b68f8430-b1b6-46d9-8358-74f04b66c9d6\") " pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:43.075361 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.075147    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c4b4d38-64d6-408a-bf2f-591822b41ac2-cert\") pod \"ingress-canary-bw662\" (UID: \"4c4b4d38-64d6-408a-bf2f-591822b41ac2\") " pod="openshift-ingress-canary/ingress-canary-bw662"
Sep 15 22:27:43.075361 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.075176    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls podName:0eea82c0-60f7-4ee7-9a1d-8bd13f06c028 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:43.575157046 +0000 UTC m=+36.193971951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls") pod "dns-default-sm96j" (UID: "0eea82c0-60f7-4ee7-9a1d-8bd13f06c028") : secret "dns-default-metrics-tls" not found
Sep 15 22:27:43.075361 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.075207    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2ba350-9c05-4b47-9f16-8b19093b9137-serving-cert\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:43.075928 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.075905    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-config-volume\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:43.075984 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.075941    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b68f8430-b1b6-46d9-8358-74f04b66c9d6-config\") pod \"console-operator-696884fb9c-9xx4f\" (UID: \"b68f8430-b1b6-46d9-8358-74f04b66c9d6\") " pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:43.076551 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.076523    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc2ba350-9c05-4b47-9f16-8b19093b9137-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:43.076644 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.076529    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fc2ba350-9c05-4b47-9f16-8b19093b9137-snapshots\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:43.076916 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.076864    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b68f8430-b1b6-46d9-8358-74f04b66c9d6-trusted-ca\") pod \"console-operator-696884fb9c-9xx4f\" (UID: \"b68f8430-b1b6-46d9-8358-74f04b66c9d6\") " pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:43.076916 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.076900    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc2ba350-9c05-4b47-9f16-8b19093b9137-tmp\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:43.079027 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.078177    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc2ba350-9c05-4b47-9f16-8b19093b9137-service-ca-bundle\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:43.079027 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.078761    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fc2ba350-9c05-4b47-9f16-8b19093b9137-serving-cert\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:43.081998 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.081508    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c4b4d38-64d6-408a-bf2f-591822b41ac2-cert\") pod \"ingress-canary-bw662\" (UID: \"4c4b4d38-64d6-408a-bf2f-591822b41ac2\") " pod="openshift-ingress-canary/ingress-canary-bw662"
Sep 15 22:27:43.082877 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.082853    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b68f8430-b1b6-46d9-8358-74f04b66c9d6-serving-cert\") pod \"console-operator-696884fb9c-9xx4f\" (UID: \"b68f8430-b1b6-46d9-8358-74f04b66c9d6\") " pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:43.084832 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.084770    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njk87\" (UniqueName: \"kubernetes.io/projected/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-kube-api-access-njk87\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:43.085186 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.085168    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9r6z\" (UniqueName: \"kubernetes.io/projected/fc2ba350-9c05-4b47-9f16-8b19093b9137-kube-api-access-s9r6z\") pod \"insights-operator-5844b856db-9x825\" (UID: \"fc2ba350-9c05-4b47-9f16-8b19093b9137\") " pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:43.085624 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.085601    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzm46\" (UniqueName: \"kubernetes.io/projected/2558ab4c-e858-4efa-a076-d49c1dcf1dd0-kube-api-access-fzm46\") pod \"network-check-source-6f8b768455-h6msr\" (UID: \"2558ab4c-e858-4efa-a076-d49c1dcf1dd0\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-h6msr"
Sep 15 22:27:43.086119 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.086098    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5drg\" (UniqueName: \"kubernetes.io/projected/4c4b4d38-64d6-408a-bf2f-591822b41ac2-kube-api-access-s5drg\") pod \"ingress-canary-bw662\" (UID: \"4c4b4d38-64d6-408a-bf2f-591822b41ac2\") " pod="openshift-ingress-canary/ingress-canary-bw662"
Sep 15 22:27:43.089654 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.089632    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwtzn\" (UniqueName: \"kubernetes.io/projected/b68f8430-b1b6-46d9-8358-74f04b66c9d6-kube-api-access-gwtzn\") pod \"console-operator-696884fb9c-9xx4f\" (UID: \"b68f8430-b1b6-46d9-8358-74f04b66c9d6\") " pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:43.099312 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.099260    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"
Sep 15 22:27:43.120446 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.120413    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5kgc5"
Sep 15 22:27:43.129435 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.129413    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"
Sep 15 22:27:43.137141 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.136999    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:27:43.144369 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.144308    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9x825"
Sep 15 22:27:43.158605 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.158568    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-h6msr"
Sep 15 22:27:43.176623 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.176347    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bw662"
Sep 15 22:27:43.216571 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.209951    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:43.280427 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.280400    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:43.397663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.397630    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc"]
Sep 15 22:27:43.401492 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:43.401463    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307013b9_cf57_4e9e_bddd_e9d51ef7c57b.slice/crio-965d1d3ae48055c97abd2e7d7a52c85989a5bf111bf6c33e85b330e6953fe34a WatchSource:0}: Error finding container 965d1d3ae48055c97abd2e7d7a52c85989a5bf111bf6c33e85b330e6953fe34a: Status 404 returned error can't find the container with id 965d1d3ae48055c97abd2e7d7a52c85989a5bf111bf6c33e85b330e6953fe34a
Sep 15 22:27:43.405246 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.405180    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5kgc5"]
Sep 15 22:27:43.408841 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:43.408816    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6ab2a4_b56d_45b9_81d8_1319abade096.slice/crio-98e5e2d17d55a084de42d9b8e3dd63b1d0e68d6f5a64602b7f406c688368264a WatchSource:0}: Error finding container 98e5e2d17d55a084de42d9b8e3dd63b1d0e68d6f5a64602b7f406c688368264a: Status 404 returned error can't find the container with id 98e5e2d17d55a084de42d9b8e3dd63b1d0e68d6f5a64602b7f406c688368264a
Sep 15 22:27:43.436716 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.436694    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9"]
Sep 15 22:27:43.441487 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:43.441461    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702fee44_5a8b_4364_a00f_957fab8661dd.slice/crio-8a576f79298bed19b498cf750ab7f7d1995e5bc0f6c11f518e3373a22b4b5db2 WatchSource:0}: Error finding container 8a576f79298bed19b498cf750ab7f7d1995e5bc0f6c11f518e3373a22b4b5db2: Status 404 returned error can't find the container with id 8a576f79298bed19b498cf750ab7f7d1995e5bc0f6c11f518e3373a22b4b5db2
Sep 15 22:27:43.466400 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.466372    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-9xx4f"]
Sep 15 22:27:43.469764 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.469741    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9x825"]
Sep 15 22:27:43.469933 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:43.469910    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68f8430_b1b6_46d9_8358_74f04b66c9d6.slice/crio-acc0ca63e411e018e0fd99ce2ba1ac0b914f9a4b1517b4b414a76cdeddf975f3 WatchSource:0}: Error finding container acc0ca63e411e018e0fd99ce2ba1ac0b914f9a4b1517b4b414a76cdeddf975f3: Status 404 returned error can't find the container with id acc0ca63e411e018e0fd99ce2ba1ac0b914f9a4b1517b4b414a76cdeddf975f3
Sep 15 22:27:43.474167 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:43.474146    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2ba350_9c05_4b47_9f16_8b19093b9137.slice/crio-bae690fcd4f0c97583de7615ea1ebbf0d36c72c3a8d5db921d58d256e05d63b5 WatchSource:0}: Error finding container bae690fcd4f0c97583de7615ea1ebbf0d36c72c3a8d5db921d58d256e05d63b5: Status 404 returned error can't find the container with id bae690fcd4f0c97583de7615ea1ebbf0d36c72c3a8d5db921d58d256e05d63b5
Sep 15 22:27:43.478978 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.478927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:43.478978 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.478974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:43.479132 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.479011    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:43.479132 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.479057    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:43.479132 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.479082    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-96r9x\" (UID: \"7f812033-940b-4211-bd1c-2eceec145d80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:27:43.479287 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.479153    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 22:27:43.479287 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.479176    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 22:27:43.479287 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.479189    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle podName:f53b8e84-9062-4265-9b5e-4d4b40ad7020 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:44.479167788 +0000 UTC m=+37.097982689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle") pod "router-default-6df89d9594-wrhv2" (UID: "f53b8e84-9062-4265-9b5e-4d4b40ad7020") : configmap references non-existent config key: service-ca.crt
Sep 15 22:27:43.479287 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.479214    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs podName:f53b8e84-9062-4265-9b5e-4d4b40ad7020 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:44.47920436 +0000 UTC m=+37.098019253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs") pod "router-default-6df89d9594-wrhv2" (UID: "f53b8e84-9062-4265-9b5e-4d4b40ad7020") : secret "router-metrics-certs-default" not found
Sep 15 22:27:43.479287 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.479231    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls podName:67b271e9-a2b8-4f33-be0b-102e434588ac nodeName:}" failed. No retries permitted until 2026-09-15 22:27:44.47922058 +0000 UTC m=+37.098035472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-gx49h" (UID: "67b271e9-a2b8-4f33-be0b-102e434588ac") : secret "cluster-monitoring-operator-tls" not found
Sep 15 22:27:43.479287 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.479236    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 22:27:43.479287 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.479236    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 22:27:43.479287 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.479246    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687df9c7d4-t72hr: secret "image-registry-tls" not found
Sep 15 22:27:43.479542 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.479299    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls podName:0fc17f4f-84da-4a36-a3bc-c874fb9296e8 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:44.479281874 +0000 UTC m=+37.098096776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls") pod "image-registry-687df9c7d4-t72hr" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8") : secret "image-registry-tls" not found
Sep 15 22:27:43.479542 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.479317    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls podName:7f812033-940b-4211-bd1c-2eceec145d80 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:44.479307947 +0000 UTC m=+37.098122845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-96r9x" (UID: "7f812033-940b-4211-bd1c-2eceec145d80") : secret "samples-operator-tls" not found
Sep 15 22:27:43.499803 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.499739    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-h6msr"]
Sep 15 22:27:43.502507 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.502463    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bw662"]
Sep 15 22:27:43.502661 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:43.502637    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2558ab4c_e858_4efa_a076_d49c1dcf1dd0.slice/crio-ef28c24e1f699b1ba9196c954da13554237fed5d99e2ca802c5c997407472b06 WatchSource:0}: Error finding container ef28c24e1f699b1ba9196c954da13554237fed5d99e2ca802c5c997407472b06: Status 404 returned error can't find the container with id ef28c24e1f699b1ba9196c954da13554237fed5d99e2ca802c5c997407472b06
Sep 15 22:27:43.505555 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:43.505534    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4b4d38_64d6_408a_bf2f_591822b41ac2.slice/crio-fb3ec9e3b0fd5bf6843b7f433c0088eb92e40b969dd44831293e78f4f42858a0 WatchSource:0}: Error finding container fb3ec9e3b0fd5bf6843b7f433c0088eb92e40b969dd44831293e78f4f42858a0: Status 404 returned error can't find the container with id fb3ec9e3b0fd5bf6843b7f433c0088eb92e40b969dd44831293e78f4f42858a0
Sep 15 22:27:43.579696 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.579660    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:43.579854 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.579830    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 22:27:43.579901 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:43.579890    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls podName:0eea82c0-60f7-4ee7-9a1d-8bd13f06c028 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:44.57987357 +0000 UTC m=+37.198688463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls") pod "dns-default-sm96j" (UID: "0eea82c0-60f7-4ee7-9a1d-8bd13f06c028") : secret "dns-default-metrics-tls" not found
Sep 15 22:27:43.892809 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.892778    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:27:43.893646 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.892827    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:27:43.893646 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.892780    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:27:43.897159 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.897138    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 22:27:43.897304 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.897138    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fznt9\""
Sep 15 22:27:43.897304 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.897169    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 22:27:43.897304 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:43.897160    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-29265\""
Sep 15 22:27:44.150219 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:44.150091    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f" event={"ID":"b68f8430-b1b6-46d9-8358-74f04b66c9d6","Type":"ContainerStarted","Data":"acc0ca63e411e018e0fd99ce2ba1ac0b914f9a4b1517b4b414a76cdeddf975f3"}
Sep 15 22:27:44.152946 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:44.152872    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9" event={"ID":"702fee44-5a8b-4364-a00f-957fab8661dd","Type":"ContainerStarted","Data":"8a576f79298bed19b498cf750ab7f7d1995e5bc0f6c11f518e3373a22b4b5db2"}
Sep 15 22:27:44.155592 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:44.155528    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bw662" event={"ID":"4c4b4d38-64d6-408a-bf2f-591822b41ac2","Type":"ContainerStarted","Data":"fb3ec9e3b0fd5bf6843b7f433c0088eb92e40b969dd44831293e78f4f42858a0"}
Sep 15 22:27:44.161639 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:44.161610    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-h6msr" event={"ID":"2558ab4c-e858-4efa-a076-d49c1dcf1dd0","Type":"ContainerStarted","Data":"ef28c24e1f699b1ba9196c954da13554237fed5d99e2ca802c5c997407472b06"}
Sep 15 22:27:44.163892 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:44.163867    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5kgc5" event={"ID":"7f6ab2a4-b56d-45b9-81d8-1319abade096","Type":"ContainerStarted","Data":"98e5e2d17d55a084de42d9b8e3dd63b1d0e68d6f5a64602b7f406c688368264a"}
Sep 15 22:27:44.167573 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:44.167503    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc" event={"ID":"307013b9-cf57-4e9e-bddd-e9d51ef7c57b","Type":"ContainerStarted","Data":"965d1d3ae48055c97abd2e7d7a52c85989a5bf111bf6c33e85b330e6953fe34a"}
Sep 15 22:27:44.171280 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:44.171210    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9x825" event={"ID":"fc2ba350-9c05-4b47-9f16-8b19093b9137","Type":"ContainerStarted","Data":"bae690fcd4f0c97583de7615ea1ebbf0d36c72c3a8d5db921d58d256e05d63b5"}
Sep 15 22:27:44.487687 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:44.487575    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:44.487687 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:44.487636    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:44.487687 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:44.487673    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:44.487945 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:44.487716    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:44.487945 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:44.487741    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-96r9x\" (UID: \"7f812033-940b-4211-bd1c-2eceec145d80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:27:44.487945 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:44.487893    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 22:27:44.488077 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:44.487960    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls podName:7f812033-940b-4211-bd1c-2eceec145d80 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:46.487939484 +0000 UTC m=+39.106754370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-96r9x" (UID: "7f812033-940b-4211-bd1c-2eceec145d80") : secret "samples-operator-tls" not found
Sep 15 22:27:44.488357 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:44.488222    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 22:27:44.488357 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:44.488299    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls podName:67b271e9-a2b8-4f33-be0b-102e434588ac nodeName:}" failed. No retries permitted until 2026-09-15 22:27:46.488282863 +0000 UTC m=+39.107097781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-gx49h" (UID: "67b271e9-a2b8-4f33-be0b-102e434588ac") : secret "cluster-monitoring-operator-tls" not found
Sep 15 22:27:44.488517 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:44.488402    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle podName:f53b8e84-9062-4265-9b5e-4d4b40ad7020 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:46.488388911 +0000 UTC m=+39.107203800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle") pod "router-default-6df89d9594-wrhv2" (UID: "f53b8e84-9062-4265-9b5e-4d4b40ad7020") : configmap references non-existent config key: service-ca.crt
Sep 15 22:27:44.488517 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:44.488473    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 22:27:44.488517 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:44.488485    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687df9c7d4-t72hr: secret "image-registry-tls" not found
Sep 15 22:27:44.488517 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:44.488518    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls podName:0fc17f4f-84da-4a36-a3bc-c874fb9296e8 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:46.488507783 +0000 UTC m=+39.107322672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls") pod "image-registry-687df9c7d4-t72hr" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8") : secret "image-registry-tls" not found
Sep 15 22:27:44.488730 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:44.488570    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 22:27:44.488730 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:44.488600    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs podName:f53b8e84-9062-4265-9b5e-4d4b40ad7020 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:46.488590962 +0000 UTC m=+39.107405847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs") pod "router-default-6df89d9594-wrhv2" (UID: "f53b8e84-9062-4265-9b5e-4d4b40ad7020") : secret "router-metrics-certs-default" not found
Sep 15 22:27:44.588854 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:44.588814    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:44.589139 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:44.589118    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 22:27:44.589209 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:44.589191    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls podName:0eea82c0-60f7-4ee7-9a1d-8bd13f06c028 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:46.58917224 +0000 UTC m=+39.207987146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls") pod "dns-default-sm96j" (UID: "0eea82c0-60f7-4ee7-9a1d-8bd13f06c028") : secret "dns-default-metrics-tls" not found
Sep 15 22:27:46.100707 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:46.100626    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:46.155210 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:46.155158    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:46.510174 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:46.510079    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:46.510174 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:46.510136    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:46.510435 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:46.510176    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:46.510435 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:46.510222    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:46.510435 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:46.510245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-96r9x\" (UID: \"7f812033-940b-4211-bd1c-2eceec145d80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:27:46.510435 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:46.510288    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 22:27:46.510435 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:46.510300    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle podName:f53b8e84-9062-4265-9b5e-4d4b40ad7020 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:50.510278759 +0000 UTC m=+43.129093659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle") pod "router-default-6df89d9594-wrhv2" (UID: "f53b8e84-9062-4265-9b5e-4d4b40ad7020") : configmap references non-existent config key: service-ca.crt
Sep 15 22:27:46.510435 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:46.510359    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 22:27:46.510435 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:46.510371    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs podName:f53b8e84-9062-4265-9b5e-4d4b40ad7020 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:50.510340235 +0000 UTC m=+43.129155136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs") pod "router-default-6df89d9594-wrhv2" (UID: "f53b8e84-9062-4265-9b5e-4d4b40ad7020") : secret "router-metrics-certs-default" not found
Sep 15 22:27:46.510435 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:46.510391    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 22:27:46.510435 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:46.510377    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 22:27:46.510435 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:46.510409    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls podName:7f812033-940b-4211-bd1c-2eceec145d80 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:50.51039732 +0000 UTC m=+43.129212209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-96r9x" (UID: "7f812033-940b-4211-bd1c-2eceec145d80") : secret "samples-operator-tls" not found
Sep 15 22:27:46.510435 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:46.510401    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687df9c7d4-t72hr: secret "image-registry-tls" not found
Sep 15 22:27:46.510435 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:46.510445    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls podName:67b271e9-a2b8-4f33-be0b-102e434588ac nodeName:}" failed. No retries permitted until 2026-09-15 22:27:50.510434946 +0000 UTC m=+43.129249836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-gx49h" (UID: "67b271e9-a2b8-4f33-be0b-102e434588ac") : secret "cluster-monitoring-operator-tls" not found
Sep 15 22:27:46.510935 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:46.510500    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls podName:0fc17f4f-84da-4a36-a3bc-c874fb9296e8 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:50.51049012 +0000 UTC m=+43.129305011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls") pod "image-registry-687df9c7d4-t72hr" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8") : secret "image-registry-tls" not found
Sep 15 22:27:46.611191 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:46.611158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:46.611374 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:46.611318    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 22:27:46.611440 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:46.611394    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls podName:0eea82c0-60f7-4ee7-9a1d-8bd13f06c028 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:50.611373617 +0000 UTC m=+43.230188507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls") pod "dns-default-sm96j" (UID: "0eea82c0-60f7-4ee7-9a1d-8bd13f06c028") : secret "dns-default-metrics-tls" not found
Sep 15 22:27:48.412055 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:48.412007    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:48.478770 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:48.478740    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:49.375881 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:49.375779    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:49.476069 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:49.476034    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:50.543851 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:50.543817    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:50.543869    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:50.543909    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:50.543953    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:50.543980    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-96r9x\" (UID: \"7f812033-940b-4211-bd1c-2eceec145d80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:50.544018    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle podName:f53b8e84-9062-4265-9b5e-4d4b40ad7020 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:58.543991195 +0000 UTC m=+51.162806120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle") pod "router-default-6df89d9594-wrhv2" (UID: "f53b8e84-9062-4265-9b5e-4d4b40ad7020") : configmap references non-existent config key: service-ca.crt
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:50.544060    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:50.544059    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:50.544119    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs podName:f53b8e84-9062-4265-9b5e-4d4b40ad7020 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:58.544102948 +0000 UTC m=+51.162917845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs") pod "router-default-6df89d9594-wrhv2" (UID: "f53b8e84-9062-4265-9b5e-4d4b40ad7020") : secret "router-metrics-certs-default" not found
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:50.544139    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:50.544148    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls podName:7f812033-940b-4211-bd1c-2eceec145d80 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:58.544135799 +0000 UTC m=+51.162950689 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-96r9x" (UID: "7f812033-940b-4211-bd1c-2eceec145d80") : secret "samples-operator-tls" not found
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:50.544156    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687df9c7d4-t72hr: secret "image-registry-tls" not found
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:50.544207    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls podName:0fc17f4f-84da-4a36-a3bc-c874fb9296e8 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:58.544195565 +0000 UTC m=+51.163010474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls") pod "image-registry-687df9c7d4-t72hr" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8") : secret "image-registry-tls" not found
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:50.544060    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 22:27:50.544345 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:50.544239    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls podName:67b271e9-a2b8-4f33-be0b-102e434588ac nodeName:}" failed. No retries permitted until 2026-09-15 22:27:58.544233547 +0000 UTC m=+51.163048447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-gx49h" (UID: "67b271e9-a2b8-4f33-be0b-102e434588ac") : secret "cluster-monitoring-operator-tls" not found
Sep 15 22:27:50.645353 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:50.645325    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:50.645500 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:50.645445    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 22:27:50.645500 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:50.645497    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls podName:0eea82c0-60f7-4ee7-9a1d-8bd13f06c028 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:58.645484404 +0000 UTC m=+51.264299303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls") pod "dns-default-sm96j" (UID: "0eea82c0-60f7-4ee7-9a1d-8bd13f06c028") : secret "dns-default-metrics-tls" not found
Sep 15 22:27:51.871342 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:51.871304    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:51.971731 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:51.971650    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:53.868376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:53.868337    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:53.968965 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:53.968873    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:54.203932 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.203476    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9x825" event={"ID":"fc2ba350-9c05-4b47-9f16-8b19093b9137","Type":"ContainerStarted","Data":"6f089477fd7f2af3b60c0078c7590880c14e7ff41ec240a0a6d733725e9891a0"}
Sep 15 22:27:54.205705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.205680    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/0.log"
Sep 15 22:27:54.205832 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.205723    2584 generic.go:356] "Generic (PLEG): container finished" podID="b68f8430-b1b6-46d9-8358-74f04b66c9d6" containerID="8c8ce8e5299566b28159cb89113ddd528a23ee598e33a32e1e81837e8ac60d7d" exitCode=255
Sep 15 22:27:54.205832 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.205799    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f" event={"ID":"b68f8430-b1b6-46d9-8358-74f04b66c9d6","Type":"ContainerDied","Data":"8c8ce8e5299566b28159cb89113ddd528a23ee598e33a32e1e81837e8ac60d7d"}
Sep 15 22:27:54.206007 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.205987    2584 scope.go:117] "RemoveContainer" containerID="8c8ce8e5299566b28159cb89113ddd528a23ee598e33a32e1e81837e8ac60d7d"
Sep 15 22:27:54.208134 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.208106    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9" event={"ID":"702fee44-5a8b-4364-a00f-957fab8661dd","Type":"ContainerStarted","Data":"43a236cfacf835efc9084548ac6551caa5abc151d756c14ec4ffed4a6aae039d"}
Sep 15 22:27:54.210021 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.210000    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bw662" event={"ID":"4c4b4d38-64d6-408a-bf2f-591822b41ac2","Type":"ContainerStarted","Data":"7694322f22933eb195580893c43578e0e6a4a9a57b43df16de7850a9f3875c84"}
Sep 15 22:27:54.211887 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.211646    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-h6msr" event={"ID":"2558ab4c-e858-4efa-a076-d49c1dcf1dd0","Type":"ContainerStarted","Data":"b95c699660c75fd026851da5dc863f5883e36243e0a6c8493575125249d4bbba"}
Sep 15 22:27:54.213301 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.213255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5kgc5" event={"ID":"7f6ab2a4-b56d-45b9-81d8-1319abade096","Type":"ContainerStarted","Data":"cecdc5af8d5277d092dd58ee350384dbcd84738299a8146f6a7bf59a20c11a10"}
Sep 15 22:27:54.214954 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.214936    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc" event={"ID":"307013b9-cf57-4e9e-bddd-e9d51ef7c57b","Type":"ContainerStarted","Data":"babecfb52b4a8727dfeb76a9f092bf09c08c89c882fc1a2648cee6ed86064e0c"}
Sep 15 22:27:54.218504 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.218480    2584 generic.go:356] "Generic (PLEG): container finished" podID="86c5e1a3-16e0-4171-a9a2-df5f8eba5318" containerID="f22b2c1579bd4222ad9894e836e1172d89ff66ec6167e33c5026f1058e34cc51" exitCode=0
Sep 15 22:27:54.218601 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.218513    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fz5x" event={"ID":"86c5e1a3-16e0-4171-a9a2-df5f8eba5318","Type":"ContainerDied","Data":"f22b2c1579bd4222ad9894e836e1172d89ff66ec6167e33c5026f1058e34cc51"}
Sep 15 22:27:54.221981 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.221935    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-9x825" podStartSLOduration=26.438867054 podStartE2EDuration="36.221886772s" totalImagesPullingTime="9.783019718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:54.22114646 +0000 UTC m=+46.839961369" watchObservedRunningTime="2026-09-15 22:27:54.221886772 +0000 UTC m=+46.840701676"
Sep 15 22:27:54.259549 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.259503    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5kgc5" podStartSLOduration=26.413007379 podStartE2EDuration="36.259482472s" totalImagesPullingTime="9.846475093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:54.239428651 +0000 UTC m=+46.858243561" watchObservedRunningTime="2026-09-15 22:27:54.259482472 +0000 UTC m=+46.878297381"
Sep 15 22:27:54.259858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.259832    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc" podStartSLOduration=26.406182894 podStartE2EDuration="36.259824755s" totalImagesPullingTime="9.853641861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:54.258220839 +0000 UTC m=+46.877035749" watchObservedRunningTime="2026-09-15 22:27:54.259824755 +0000 UTC m=+46.878639664"
Sep 15 22:27:54.309396 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.308330    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bw662" podStartSLOduration=2.558610608 podStartE2EDuration="12.308311124s" totalImagesPullingTime="9.749700516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:54.307785562 +0000 UTC m=+46.926600470" watchObservedRunningTime="2026-09-15 22:27:54.308311124 +0000 UTC m=+46.927126031"
Sep 15 22:27:54.352534 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.344796    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9" podStartSLOduration=26.531714163 podStartE2EDuration="36.344775975s" totalImagesPullingTime="9.813061812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:54.332116424 +0000 UTC m=+46.950931333" watchObservedRunningTime="2026-09-15 22:27:54.344775975 +0000 UTC m=+46.963590886"
Sep 15 22:27:54.352534 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.349086    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:54.371178 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.370999    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-h6msr" podStartSLOduration=11.609041162 podStartE2EDuration="21.370980102s" totalImagesPullingTime="9.76193894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:54.370927064 +0000 UTC m=+46.989741974" watchObservedRunningTime="2026-09-15 22:27:54.370980102 +0000 UTC m=+46.989795010"
Sep 15 22:27:54.439872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.439843    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:54.897888 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.897849    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b46hw"]
Sep 15 22:27:54.924940 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.924908    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:54.930443 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.930422    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b46hw"]
Sep 15 22:27:54.930576 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.930557    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:54.933834 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.933194    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 22:27:54.933834 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.933488    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-44p4v\""
Sep 15 22:27:54.933834 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.933681    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 22:27:54.984094 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.984065    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:54.984231 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.984106    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-crio-socket\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:54.984231 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.984126    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-data-volume\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:54.984324 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.984255    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:54.984324 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:54.984303    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pplvn\" (UniqueName: \"kubernetes.io/projected/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-api-access-pplvn\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:55.000095 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.000067    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-xdpxh"]
Sep 15 22:27:55.012434 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.012406    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:55.023542 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.023524    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-xdpxh"]
Sep 15 22:27:55.023647 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.023611    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xdpxh"
Sep 15 22:27:55.026362 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.026341    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 15 22:27:55.026512 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.026367    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-z4vcl\""
Sep 15 22:27:55.027049 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.027029    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 15 22:27:55.080422 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.080387    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:55.085358 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.085338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-crio-socket\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:55.085471 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.085364    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-data-volume\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:55.085551 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.085533    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:55.085605 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.085568    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pplvn\" (UniqueName: \"kubernetes.io/projected/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-api-access-pplvn\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:55.085605 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.085577    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-crio-socket\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:55.085689 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.085641    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:55.085689 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.085653    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-data-volume\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:55.085689 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.085663    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjjrr\" (UniqueName: \"kubernetes.io/projected/34d78629-cec9-4ed2-a073-5152edc9dec7-kube-api-access-xjjrr\") pod \"migrator-8b4d697c5-xdpxh\" (UID: \"34d78629-cec9-4ed2-a073-5152edc9dec7\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xdpxh"
Sep 15 22:27:55.085689 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:55.085673    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 22:27:55.085858 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:55.085713    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 22:27:55.085858 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:55.085734    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls podName:17be5533-4f9e-4bd3-9198-48f1cb3e8ba8 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:55.585715648 +0000 UTC m=+48.204530552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b46hw" (UID: "17be5533-4f9e-4bd3-9198-48f1cb3e8ba8") : secret "insights-runtime-extractor-tls" not found
Sep 15 22:27:55.085858 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:55.085753    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-rbac-proxy-cm podName:17be5533-4f9e-4bd3-9198-48f1cb3e8ba8 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:55.585743655 +0000 UTC m=+48.204558545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b46hw" (UID: "17be5533-4f9e-4bd3-9198-48f1cb3e8ba8") : configmap "kube-rbac-proxy" not found
Sep 15 22:27:55.095048 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.095029    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pplvn\" (UniqueName: \"kubernetes.io/projected/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-api-access-pplvn\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:55.187104 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.187041    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjjrr\" (UniqueName: \"kubernetes.io/projected/34d78629-cec9-4ed2-a073-5152edc9dec7-kube-api-access-xjjrr\") pod \"migrator-8b4d697c5-xdpxh\" (UID: \"34d78629-cec9-4ed2-a073-5152edc9dec7\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xdpxh"
Sep 15 22:27:55.194640 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.194618    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjjrr\" (UniqueName: \"kubernetes.io/projected/34d78629-cec9-4ed2-a073-5152edc9dec7-kube-api-access-xjjrr\") pod \"migrator-8b4d697c5-xdpxh\" (UID: \"34d78629-cec9-4ed2-a073-5152edc9dec7\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xdpxh"
Sep 15 22:27:55.225911 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.225883    2584 generic.go:356] "Generic (PLEG): container finished" podID="86c5e1a3-16e0-4171-a9a2-df5f8eba5318" containerID="f8751b4b0ae6d5702f97eee3a78d2c83f28e0ac3ec9985beaa20ff2361ab053e" exitCode=0
Sep 15 22:27:55.225996 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.225948    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fz5x" event={"ID":"86c5e1a3-16e0-4171-a9a2-df5f8eba5318","Type":"ContainerDied","Data":"f8751b4b0ae6d5702f97eee3a78d2c83f28e0ac3ec9985beaa20ff2361ab053e"}
Sep 15 22:27:55.227707 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.227689    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:27:55.228134 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.228114    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/0.log"
Sep 15 22:27:55.228202 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.228154    2584 generic.go:356] "Generic (PLEG): container finished" podID="b68f8430-b1b6-46d9-8358-74f04b66c9d6" containerID="1bb0085eb6b397a4cf1295ee038fb0267737f5fc6bf4a8e12526eb416c6ae5b5" exitCode=255
Sep 15 22:27:55.228237 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.228216    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f" event={"ID":"b68f8430-b1b6-46d9-8358-74f04b66c9d6","Type":"ContainerDied","Data":"1bb0085eb6b397a4cf1295ee038fb0267737f5fc6bf4a8e12526eb416c6ae5b5"}
Sep 15 22:27:55.228300 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.228256    2584 scope.go:117] "RemoveContainer" containerID="8c8ce8e5299566b28159cb89113ddd528a23ee598e33a32e1e81837e8ac60d7d"
Sep 15 22:27:55.228417 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.228402    2584 scope.go:117] "RemoveContainer" containerID="1bb0085eb6b397a4cf1295ee038fb0267737f5fc6bf4a8e12526eb416c6ae5b5"
Sep 15 22:27:55.228558 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:55.228539    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-9xx4f_openshift-console-operator(b68f8430-b1b6-46d9-8358-74f04b66c9d6)\"" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f" podUID="b68f8430-b1b6-46d9-8358-74f04b66c9d6"
Sep 15 22:27:55.332236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.332003    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xdpxh"
Sep 15 22:27:55.480386 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.480312    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:55.538366 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.538333    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:55.590592 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.590571    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:55.590710 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.590657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:55.590710 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:55.590700    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 22:27:55.590818 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:55.590777    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-rbac-proxy-cm podName:17be5533-4f9e-4bd3-9198-48f1cb3e8ba8 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:56.590754009 +0000 UTC m=+49.209568900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b46hw" (UID: "17be5533-4f9e-4bd3-9198-48f1cb3e8ba8") : configmap "kube-rbac-proxy" not found
Sep 15 22:27:55.590880 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:55.590816    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 22:27:55.590880 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:55.590865    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls podName:17be5533-4f9e-4bd3-9198-48f1cb3e8ba8 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:56.590849802 +0000 UTC m=+49.209664701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b46hw" (UID: "17be5533-4f9e-4bd3-9198-48f1cb3e8ba8") : secret "insights-runtime-extractor-tls" not found
Sep 15 22:27:55.620689 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.620650    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-xdpxh"]
Sep 15 22:27:55.623564 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:55.623541    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d78629_cec9_4ed2_a073_5152edc9dec7.slice/crio-ef76189eeceb01c94c565c2da099c1bd45fd67544011803b3fe7c47fac9fd76e WatchSource:0}: Error finding container ef76189eeceb01c94c565c2da099c1bd45fd67544011803b3fe7c47fac9fd76e: Status 404 returned error can't find the container with id ef76189eeceb01c94c565c2da099c1bd45fd67544011803b3fe7c47fac9fd76e
Sep 15 22:27:55.877699 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.877676    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dwfjr_e3deae5a-e1be-4af6-8ac0-ef4976250510/dns-node-resolver/0.log"
Sep 15 22:27:55.916372 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:55.916331    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:56.194374 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:56.194302    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:56.233702 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:56.233677    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:27:56.233993 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:56.233977    2584 scope.go:117] "RemoveContainer" containerID="1bb0085eb6b397a4cf1295ee038fb0267737f5fc6bf4a8e12526eb416c6ae5b5"
Sep 15 22:27:56.234155 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:56.234134    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-9xx4f_openshift-console-operator(b68f8430-b1b6-46d9-8358-74f04b66c9d6)\"" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f" podUID="b68f8430-b1b6-46d9-8358-74f04b66c9d6"
Sep 15 22:27:56.234970 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:56.234940    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xdpxh" event={"ID":"34d78629-cec9-4ed2-a073-5152edc9dec7","Type":"ContainerStarted","Data":"ef76189eeceb01c94c565c2da099c1bd45fd67544011803b3fe7c47fac9fd76e"}
Sep 15 22:27:56.238550 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:56.238531    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fz5x" event={"ID":"86c5e1a3-16e0-4171-a9a2-df5f8eba5318","Type":"ContainerStarted","Data":"dc328c605c32e051b6f40f242e9fa6b3b2e8c96e90637ff81bcac49455ef1e31"}
Sep 15 22:27:56.254299 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:56.254254    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:56.272125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:56.272085    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5fz5x" podStartSLOduration=10.071402776 podStartE2EDuration="48.272068394s" totalImagesPullingTime="38.200665618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:56.271183986 +0000 UTC m=+48.889998893" watchObservedRunningTime="2026-09-15 22:27:56.272068394 +0000 UTC m=+48.890883303"
Sep 15 22:27:56.498714 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:56.498638    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:56.594094 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:56.594058    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:56.599054 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:56.599035    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:56.599155 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:56.599141    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 22:27:56.599203 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:56.599194    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-rbac-proxy-cm podName:17be5533-4f9e-4bd3-9198-48f1cb3e8ba8 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:58.599179881 +0000 UTC m=+51.217994766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b46hw" (UID: "17be5533-4f9e-4bd3-9198-48f1cb3e8ba8") : configmap "kube-rbac-proxy" not found
Sep 15 22:27:56.599241 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:56.599140    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:56.599241 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:56.599211    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 22:27:56.599321 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:56.599260    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls podName:17be5533-4f9e-4bd3-9198-48f1cb3e8ba8 nodeName:}" failed. No retries permitted until 2026-09-15 22:27:58.599248852 +0000 UTC m=+51.218063751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b46hw" (UID: "17be5533-4f9e-4bd3-9198-48f1cb3e8ba8") : secret "insights-runtime-extractor-tls" not found
Sep 15 22:27:57.074193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.074159    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8jwbj_c8e67238-6768-4b3c-89f2-a17e61edee60/node-ca/0.log"
Sep 15 22:27:57.673525 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.673467    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bw662_4c4b4d38-64d6-408a-bf2f-591822b41ac2/serve-healthcheck-canary/0.log"
Sep 15 22:27:57.791585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.791554    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-7w8fz"]
Sep 15 22:27:57.807712 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.807691    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-7w8fz"]
Sep 15 22:27:57.807837 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.807810    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-7w8fz"
Sep 15 22:27:57.810381 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.810360    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dss5j\""
Sep 15 22:27:57.810628 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.810602    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 15 22:27:57.811705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.811686    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 15 22:27:57.811828 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.811735    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 15 22:27:57.811828 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.811744    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 15 22:27:57.817833 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.817813    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:57.875861 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.875837    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:57.908855 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.908832    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5e3d250e-d387-45fb-a1f2-e35f0937e09b-signing-cabundle\") pod \"service-ca-568869fdc5-7w8fz\" (UID: \"5e3d250e-d387-45fb-a1f2-e35f0937e09b\") " pod="openshift-service-ca/service-ca-568869fdc5-7w8fz"
Sep 15 22:27:57.908978 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.908892    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9cz2\" (UniqueName: \"kubernetes.io/projected/5e3d250e-d387-45fb-a1f2-e35f0937e09b-kube-api-access-d9cz2\") pod \"service-ca-568869fdc5-7w8fz\" (UID: \"5e3d250e-d387-45fb-a1f2-e35f0937e09b\") " pod="openshift-service-ca/service-ca-568869fdc5-7w8fz"
Sep 15 22:27:57.908978 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:57.908934    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5e3d250e-d387-45fb-a1f2-e35f0937e09b-signing-key\") pod \"service-ca-568869fdc5-7w8fz\" (UID: \"5e3d250e-d387-45fb-a1f2-e35f0937e09b\") " pod="openshift-service-ca/service-ca-568869fdc5-7w8fz"
Sep 15 22:27:58.010194 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.010133    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5e3d250e-d387-45fb-a1f2-e35f0937e09b-signing-cabundle\") pod \"service-ca-568869fdc5-7w8fz\" (UID: \"5e3d250e-d387-45fb-a1f2-e35f0937e09b\") " pod="openshift-service-ca/service-ca-568869fdc5-7w8fz"
Sep 15 22:27:58.010338 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.010240    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9cz2\" (UniqueName: \"kubernetes.io/projected/5e3d250e-d387-45fb-a1f2-e35f0937e09b-kube-api-access-d9cz2\") pod \"service-ca-568869fdc5-7w8fz\" (UID: \"5e3d250e-d387-45fb-a1f2-e35f0937e09b\") " pod="openshift-service-ca/service-ca-568869fdc5-7w8fz"
Sep 15 22:27:58.010338 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.010299    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5e3d250e-d387-45fb-a1f2-e35f0937e09b-signing-key\") pod \"service-ca-568869fdc5-7w8fz\" (UID: \"5e3d250e-d387-45fb-a1f2-e35f0937e09b\") " pod="openshift-service-ca/service-ca-568869fdc5-7w8fz"
Sep 15 22:27:58.011613 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.011104    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5e3d250e-d387-45fb-a1f2-e35f0937e09b-signing-cabundle\") pod \"service-ca-568869fdc5-7w8fz\" (UID: \"5e3d250e-d387-45fb-a1f2-e35f0937e09b\") " pod="openshift-service-ca/service-ca-568869fdc5-7w8fz"
Sep 15 22:27:58.014586 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.013521    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5e3d250e-d387-45fb-a1f2-e35f0937e09b-signing-key\") pod \"service-ca-568869fdc5-7w8fz\" (UID: \"5e3d250e-d387-45fb-a1f2-e35f0937e09b\") " pod="openshift-service-ca/service-ca-568869fdc5-7w8fz"
Sep 15 22:27:58.018987 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.018967    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9cz2\" (UniqueName: \"kubernetes.io/projected/5e3d250e-d387-45fb-a1f2-e35f0937e09b-kube-api-access-d9cz2\") pod \"service-ca-568869fdc5-7w8fz\" (UID: \"5e3d250e-d387-45fb-a1f2-e35f0937e09b\") " pod="openshift-service-ca/service-ca-568869fdc5-7w8fz"
Sep 15 22:27:58.117240 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.117211    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-7w8fz"
Sep 15 22:27:58.246757 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.246718    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xdpxh" event={"ID":"34d78629-cec9-4ed2-a073-5152edc9dec7","Type":"ContainerStarted","Data":"aa6479250ab6b7be7ec8fba8e82b74ca0f84bf8cc3b55e67107f321306671422"}
Sep 15 22:27:58.246912 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.246814    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xdpxh" event={"ID":"34d78629-cec9-4ed2-a073-5152edc9dec7","Type":"ContainerStarted","Data":"1bc6b7be4d82e3ea4b1a9281185516e62f885dde7f00ebb045e3263df602847e"}
Sep 15 22:27:58.263152 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.263071    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xdpxh" podStartSLOduration=2.62475632 podStartE2EDuration="4.263057034s" totalImagesPullingTime="1.638300714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:58.262656512 +0000 UTC m=+50.881471420" watchObservedRunningTime="2026-09-15 22:27:58.263057034 +0000 UTC m=+50.881871941"
Sep 15 22:27:58.355146 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.355111    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:58.411559 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.411493    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-7w8fz"]
Sep 15 22:27:58.414813 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:27:58.414791    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3d250e_d387_45fb_a1f2_e35f0937e09b.slice/crio-a76e97c33f83fceb9f92059c5b86e07d8c938f1c284a819aa054cff30d48bef5 WatchSource:0}: Error finding container a76e97c33f83fceb9f92059c5b86e07d8c938f1c284a819aa054cff30d48bef5: Status 404 returned error can't find the container with id a76e97c33f83fceb9f92059c5b86e07d8c938f1c284a819aa054cff30d48bef5
Sep 15 22:27:58.456691 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.456665    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:58.615280 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.615232    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:27:58.615422 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.615299    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-96r9x\" (UID: \"7f812033-940b-4211-bd1c-2eceec145d80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:27:58.615422 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.615338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:58.615422 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.615373    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 22:27:58.615422 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.615392    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687df9c7d4-t72hr: secret "image-registry-tls" not found
Sep 15 22:27:58.615583 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.615433    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 22:27:58.615583 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.615438    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls podName:0fc17f4f-84da-4a36-a3bc-c874fb9296e8 nodeName:}" failed. No retries permitted until 2026-09-15 22:28:14.615423103 +0000 UTC m=+67.234237989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls") pod "image-registry-687df9c7d4-t72hr" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8") : secret "image-registry-tls" not found
Sep 15 22:27:58.615583 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.615491    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls podName:7f812033-940b-4211-bd1c-2eceec145d80 nodeName:}" failed. No retries permitted until 2026-09-15 22:28:14.615477877 +0000 UTC m=+67.234292763 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-96r9x" (UID: "7f812033-940b-4211-bd1c-2eceec145d80") : secret "samples-operator-tls" not found
Sep 15 22:27:58.615583 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.615499    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 22:27:58.615583 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.615543    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls podName:17be5533-4f9e-4bd3-9198-48f1cb3e8ba8 nodeName:}" failed. No retries permitted until 2026-09-15 22:28:02.615529271 +0000 UTC m=+55.234344171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b46hw" (UID: "17be5533-4f9e-4bd3-9198-48f1cb3e8ba8") : secret "insights-runtime-extractor-tls" not found
Sep 15 22:27:58.615793 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.615613    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:58.615793 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.615642    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:58.615793 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.615681    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:27:58.615793 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.615717    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:27:58.615793 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.615783    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 22:27:58.616001 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.615821    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs podName:f53b8e84-9062-4265-9b5e-4d4b40ad7020 nodeName:}" failed. No retries permitted until 2026-09-15 22:28:14.615813595 +0000 UTC m=+67.234628480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs") pod "router-default-6df89d9594-wrhv2" (UID: "f53b8e84-9062-4265-9b5e-4d4b40ad7020") : secret "router-metrics-certs-default" not found
Sep 15 22:27:58.616001 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.615820    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 22:27:58.616001 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.615832    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle podName:f53b8e84-9062-4265-9b5e-4d4b40ad7020 nodeName:}" failed. No retries permitted until 2026-09-15 22:28:14.615826486 +0000 UTC m=+67.234641372 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle") pod "router-default-6df89d9594-wrhv2" (UID: "f53b8e84-9062-4265-9b5e-4d4b40ad7020") : configmap references non-existent config key: service-ca.crt
Sep 15 22:27:58.616001 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.615857    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls podName:67b271e9-a2b8-4f33-be0b-102e434588ac nodeName:}" failed. No retries permitted until 2026-09-15 22:28:14.615846397 +0000 UTC m=+67.234661306 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-gx49h" (UID: "67b271e9-a2b8-4f33-be0b-102e434588ac") : secret "cluster-monitoring-operator-tls" not found
Sep 15 22:27:58.616127 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.616054    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:27:58.716226 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.716197    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:27:58.716393 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.716328    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 22:27:58.716393 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:27:58.716384    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls podName:0eea82c0-60f7-4ee7-9a1d-8bd13f06c028 nodeName:}" failed. No retries permitted until 2026-09-15 22:28:14.716368052 +0000 UTC m=+67.335182938 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls") pod "dns-default-sm96j" (UID: "0eea82c0-60f7-4ee7-9a1d-8bd13f06c028") : secret "dns-default-metrics-tls" not found
Sep 15 22:27:58.804066 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:58.804036    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:59.251697 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:59.251624    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-7w8fz" event={"ID":"5e3d250e-d387-45fb-a1f2-e35f0937e09b","Type":"ContainerStarted","Data":"6572595ffbdac9ae24237ab266709bf2318e3c69089e0be606d2dd63e48dc925"}
Sep 15 22:27:59.251697 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:59.251675    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-7w8fz" event={"ID":"5e3d250e-d387-45fb-a1f2-e35f0937e09b","Type":"ContainerStarted","Data":"a76e97c33f83fceb9f92059c5b86e07d8c938f1c284a819aa054cff30d48bef5"}
Sep 15 22:27:59.273226 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:59.273173    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-7w8fz" podStartSLOduration=2.273155494 podStartE2EDuration="2.273155494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:27:59.271128334 +0000 UTC m=+51.889943241" watchObservedRunningTime="2026-09-15 22:27:59.273155494 +0000 UTC m=+51.891970394"
Sep 15 22:27:59.696928 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:59.696884    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:27:59.956904 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:27:59.956817    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:00.053586 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:00.053553    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:00.167505 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:00.167475    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:00.268810 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:00.268711    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:00.404171 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:00.404136    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:01.193283 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:01.193238    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:01.396189 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:01.396155    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:01.989390 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:01.989355    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:02.398098 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:02.398064    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:02.655596 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:02.655509    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:28:02.655761 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:28:02.655694    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 22:28:02.655826 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:28:02.655775    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls podName:17be5533-4f9e-4bd3-9198-48f1cb3e8ba8 nodeName:}" failed. No retries permitted until 2026-09-15 22:28:10.655752787 +0000 UTC m=+63.274567680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b46hw" (UID: "17be5533-4f9e-4bd3-9198-48f1cb3e8ba8") : secret "insights-runtime-extractor-tls" not found
Sep 15 22:28:02.662543 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:02.662511    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:02.826611 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:02.826571    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:03.020088 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:03.019989    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:03.137407 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:03.137380    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:28:03.137556 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:03.137456    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:28:03.137776 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:03.137765    2584 scope.go:117] "RemoveContainer" containerID="1bb0085eb6b397a4cf1295ee038fb0267737f5fc6bf4a8e12526eb416c6ae5b5"
Sep 15 22:28:03.137921 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:28:03.137904    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-9xx4f_openshift-console-operator(b68f8430-b1b6-46d9-8358-74f04b66c9d6)\"" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f" podUID="b68f8430-b1b6-46d9-8358-74f04b66c9d6"
Sep 15 22:28:03.419630 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:03.419593    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:03.561246 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:03.561214    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:03.803653 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:03.803623    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:04.198952 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:04.198864    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:04.392239 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:04.392192    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:04.495318 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:04.495231    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:04.817061 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:04.817023    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:05.035787 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:05.035751    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:05.419141 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:05.419104    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:05.637920 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:05.637888    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:06.207094 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:06.207061    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:06.386895 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:06.386869    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:06.789203 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:06.789166    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:06.983283 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:06.983243    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:07.138522 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:07.138496    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hd5sn"
Sep 15 22:28:07.591689 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:07.591642    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:08.187289 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:08.187232    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:08.391599 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:08.391559    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:08.798951 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:08.798921    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:08.989524 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:08.989487    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:09.389164 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:09.389132    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:09.561329 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:09.561293    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:09.783246 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:09.783163    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:10.090485 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:10.090452    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:10.190823 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:10.190792    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:10.402578 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:10.402490    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:10.715456 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:10.715368    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:28:10.717654 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:10.717631    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17be5533-4f9e-4bd3-9198-48f1cb3e8ba8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b46hw\" (UID: \"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8\") " pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:28:10.843759 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:10.843733    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-44p4v\""
Sep 15 22:28:10.851941 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:10.851923    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b46hw"
Sep 15 22:28:11.087595 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:11.087568    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:11.129402 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:11.129381    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b46hw"]
Sep 15 22:28:11.131688 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:28:11.131661    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17be5533_4f9e_4bd3_9198_48f1cb3e8ba8.slice/crio-304ac2a441dba159d88cd217700ff303dd9a56bb930342b1fe4ae5a3185696ed WatchSource:0}: Error finding container 304ac2a441dba159d88cd217700ff303dd9a56bb930342b1fe4ae5a3185696ed: Status 404 returned error can't find the container with id 304ac2a441dba159d88cd217700ff303dd9a56bb930342b1fe4ae5a3185696ed
Sep 15 22:28:11.187324 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:11.187294    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:11.292164 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:11.292130    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b46hw" event={"ID":"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8","Type":"ContainerStarted","Data":"6a108150c72b93fbe7282e9cd1305c1e32ef2ac9eb03464b684e5eaace73a770"}
Sep 15 22:28:11.292343 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:11.292196    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b46hw" event={"ID":"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8","Type":"ContainerStarted","Data":"304ac2a441dba159d88cd217700ff303dd9a56bb930342b1fe4ae5a3185696ed"}
Sep 15 22:28:12.296970 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:12.296940    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b46hw" event={"ID":"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8","Type":"ContainerStarted","Data":"de66e9eb23fe346cb75f56098790539f76b0ab08a667925e3294a12365ae546e"}
Sep 15 22:28:13.647628 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:13.647553    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs\") pod \"network-metrics-daemon-pc4cx\" (UID: \"e8d2b3b9-6751-4c72-9677-dabc86c07f2a\") " pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:28:13.650561 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:13.650543    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 22:28:13.660130 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:13.660102    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d2b3b9-6751-4c72-9677-dabc86c07f2a-metrics-certs\") pod \"network-metrics-daemon-pc4cx\" (UID: \"e8d2b3b9-6751-4c72-9677-dabc86c07f2a\") " pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:28:13.748081 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:13.748042    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:28:13.751024 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:13.751009    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 22:28:13.760290 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:13.760249    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c-original-pull-secret\") pod \"global-pull-secret-syncer-mn2ls\" (UID: \"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c\") " pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:28:13.848773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:13.848737    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8ds\" (UniqueName: \"kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds\") pod \"network-check-target-jc5qb\" (UID: \"752e7fb0-13a9-4aee-b255-eb0eb8c04bdf\") " pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:28:13.851068 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:13.851050    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr8ds\" (UniqueName: \"kubernetes.io/projected/752e7fb0-13a9-4aee-b255-eb0eb8c04bdf-kube-api-access-rr8ds\") pod \"network-check-target-jc5qb\" (UID: \"752e7fb0-13a9-4aee-b255-eb0eb8c04bdf\") " pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:28:13.908710 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:13.908654    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fznt9\""
Sep 15 22:28:13.916261 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:13.916242    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:28:13.916530 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:13.916516    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-29265\""
Sep 15 22:28:13.920827 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:13.920813    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mn2ls"
Sep 15 22:28:13.924963 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:13.924862    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pc4cx"
Sep 15 22:28:14.191904 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.191766    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jc5qb"]
Sep 15 22:28:14.194514 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:28:14.194489    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752e7fb0_13a9_4aee_b255_eb0eb8c04bdf.slice/crio-94a5dea1fc21acdbe5c5d57b867042f949e905e6689650d57de0c989fe385994 WatchSource:0}: Error finding container 94a5dea1fc21acdbe5c5d57b867042f949e905e6689650d57de0c989fe385994: Status 404 returned error can't find the container with id 94a5dea1fc21acdbe5c5d57b867042f949e905e6689650d57de0c989fe385994
Sep 15 22:28:14.194643 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.194556    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mn2ls"]
Sep 15 22:28:14.204373 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.204349    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pc4cx"]
Sep 15 22:28:14.304958 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.304925    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b46hw" event={"ID":"17be5533-4f9e-4bd3-9198-48f1cb3e8ba8","Type":"ContainerStarted","Data":"1073f89de0bbb8ad31ef77e587449c32f2b8d56dc5502eac426f203f9d6ddd30"}
Sep 15 22:28:14.306154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.306128    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pc4cx" event={"ID":"e8d2b3b9-6751-4c72-9677-dabc86c07f2a","Type":"ContainerStarted","Data":"9f13ddd419ef015feda2731c5d265bbb6cdf20f7870315662f26f3262936cb3f"}
Sep 15 22:28:14.307233 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.307213    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mn2ls" event={"ID":"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c","Type":"ContainerStarted","Data":"44c8b45e105df5d7d09b6dbdd2868b7d5a0931deb78e60add76796a6fc4b9dc0"}
Sep 15 22:28:14.308785 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.308765    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jc5qb" event={"ID":"752e7fb0-13a9-4aee-b255-eb0eb8c04bdf","Type":"ContainerStarted","Data":"b57b765488393cdafe8cc59cd7828b3e53a0ed476fd0d9bdc05866371ed2e462"}
Sep 15 22:28:14.308874 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.308797    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jc5qb" event={"ID":"752e7fb0-13a9-4aee-b255-eb0eb8c04bdf","Type":"ContainerStarted","Data":"94a5dea1fc21acdbe5c5d57b867042f949e905e6689650d57de0c989fe385994"}
Sep 15 22:28:14.308945 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.308903    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:28:14.326878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.326839    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b46hw" podStartSLOduration=18.27176327 podStartE2EDuration="20.326826702s" totalImagesPullingTime="2.055063432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:27:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:14.325878646 +0000 UTC m=+66.944693554" watchObservedRunningTime="2026-09-15 22:28:14.326826702 +0000 UTC m=+66.945641611"
Sep 15 22:28:14.341488 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.341450    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jc5qb" podStartSLOduration=66.341439576 podStartE2EDuration="1m6.341439576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:14.340378968 +0000 UTC m=+66.959193877" watchObservedRunningTime="2026-09-15 22:28:14.341439576 +0000 UTC m=+66.960254483"
Sep 15 22:28:14.656335 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.656305    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:28:14.656335 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.656335    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-96r9x\" (UID: \"7f812033-940b-4211-bd1c-2eceec145d80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:28:14.656800 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.656384    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:28:14.656800 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.656405    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:28:14.656800 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.656425    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:28:14.657153 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.657129    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8e84-9062-4265-9b5e-4d4b40ad7020-service-ca-bundle\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:28:14.658712 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.658681    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b271e9-a2b8-4f33-be0b-102e434588ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-gx49h\" (UID: \"67b271e9-a2b8-4f33-be0b-102e434588ac\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:28:14.658823 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.658800    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53b8e84-9062-4265-9b5e-4d4b40ad7020-metrics-certs\") pod \"router-default-6df89d9594-wrhv2\" (UID: \"f53b8e84-9062-4265-9b5e-4d4b40ad7020\") " pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:28:14.659154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.659132    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls\") pod \"image-registry-687df9c7d4-t72hr\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") " pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:28:14.659193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.659132    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7f812033-940b-4211-bd1c-2eceec145d80-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-96r9x\" (UID: \"7f812033-940b-4211-bd1c-2eceec145d80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:28:14.757356 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.757329    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:28:14.759255 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.759238    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0eea82c0-60f7-4ee7-9a1d-8bd13f06c028-metrics-tls\") pod \"dns-default-sm96j\" (UID: \"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028\") " pod="openshift-dns/dns-default-sm96j"
Sep 15 22:28:14.872016 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.871991    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-28xs6\""
Sep 15 22:28:14.878698 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.878671    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6tvss\""
Sep 15 22:28:14.879722 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.879704    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"
Sep 15 22:28:14.887431 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.887405    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:28:14.896119 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.896098    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dzgwm\""
Sep 15 22:28:14.903734 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.903664    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:28:14.911442 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.911201    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mwpvf\""
Sep 15 22:28:14.918737 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.918713    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"
Sep 15 22:28:14.956392 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.954856    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8pmw9\""
Sep 15 22:28:14.964012 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:14.963753    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sm96j"
Sep 15 22:28:15.165355 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:15.165246    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h"]
Sep 15 22:28:15.167753 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:28:15.167718    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b271e9_a2b8_4f33_be0b_102e434588ac.slice/crio-93db5d401b511bf0aa15e5bef13c12ff5a5c1a8c493698b82034aacfa3858716 WatchSource:0}: Error finding container 93db5d401b511bf0aa15e5bef13c12ff5a5c1a8c493698b82034aacfa3858716: Status 404 returned error can't find the container with id 93db5d401b511bf0aa15e5bef13c12ff5a5c1a8c493698b82034aacfa3858716
Sep 15 22:28:15.169302 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:15.169280    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-687df9c7d4-t72hr"]
Sep 15 22:28:15.172183 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:28:15.172154    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc17f4f_84da_4a36_a3bc_c874fb9296e8.slice/crio-a571bc26c9c572fe0710b8ac3da5ede6eb8ece02fe3a71bad57e789a329ff252 WatchSource:0}: Error finding container a571bc26c9c572fe0710b8ac3da5ede6eb8ece02fe3a71bad57e789a329ff252: Status 404 returned error can't find the container with id a571bc26c9c572fe0710b8ac3da5ede6eb8ece02fe3a71bad57e789a329ff252
Sep 15 22:28:15.206325 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:15.206297    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6df89d9594-wrhv2"]
Sep 15 22:28:15.210432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:15.210365    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x"]
Sep 15 22:28:15.259027 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:15.258979    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sm96j"]
Sep 15 22:28:15.315869 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:15.315832    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687df9c7d4-t72hr" event={"ID":"0fc17f4f-84da-4a36-a3bc-c874fb9296e8","Type":"ContainerStarted","Data":"a4531a5bd6a1f4215c5334d0d743bc2cc4f543b146f9c193ef276c77d796d93a"}
Sep 15 22:28:15.316030 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:15.315894    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687df9c7d4-t72hr" event={"ID":"0fc17f4f-84da-4a36-a3bc-c874fb9296e8","Type":"ContainerStarted","Data":"a571bc26c9c572fe0710b8ac3da5ede6eb8ece02fe3a71bad57e789a329ff252"}
Sep 15 22:28:15.316645 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:15.316622    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:28:15.318310 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:15.318284    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h" event={"ID":"67b271e9-a2b8-4f33-be0b-102e434588ac","Type":"ContainerStarted","Data":"93db5d401b511bf0aa15e5bef13c12ff5a5c1a8c493698b82034aacfa3858716"}
Sep 15 22:28:15.345663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:15.345346    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-687df9c7d4-t72hr" podStartSLOduration=67.345325398 podStartE2EDuration="1m7.345325398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:15.344519074 +0000 UTC m=+67.963333982" watchObservedRunningTime="2026-09-15 22:28:15.345325398 +0000 UTC m=+67.964140306"
Sep 15 22:28:15.446537 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:28:15.446468    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eea82c0_60f7_4ee7_9a1d_8bd13f06c028.slice/crio-1f82313f86a78a94afda15af92cdd8c442c7de29a65eafa64c9a721526767551 WatchSource:0}: Error finding container 1f82313f86a78a94afda15af92cdd8c442c7de29a65eafa64c9a721526767551: Status 404 returned error can't find the container with id 1f82313f86a78a94afda15af92cdd8c442c7de29a65eafa64c9a721526767551
Sep 15 22:28:15.447520 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:28:15.447479    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53b8e84_9062_4265_9b5e_4d4b40ad7020.slice/crio-9577eadd279713e67ba4d1eb85aacaaef1bc4d978457b80879c8aad61e8a6a47 WatchSource:0}: Error finding container 9577eadd279713e67ba4d1eb85aacaaef1bc4d978457b80879c8aad61e8a6a47: Status 404 returned error can't find the container with id 9577eadd279713e67ba4d1eb85aacaaef1bc4d978457b80879c8aad61e8a6a47
Sep 15 22:28:15.892861 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:15.892828    2584 scope.go:117] "RemoveContainer" containerID="1bb0085eb6b397a4cf1295ee038fb0267737f5fc6bf4a8e12526eb416c6ae5b5"
Sep 15 22:28:16.333993 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.333399    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6df89d9594-wrhv2" event={"ID":"f53b8e84-9062-4265-9b5e-4d4b40ad7020","Type":"ContainerStarted","Data":"d9c2cebda6b44aa4fbb4e6fb49fd6111183b7a7e5b3d92d77048eacc11c3ae43"}
Sep 15 22:28:16.333993 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.333456    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6df89d9594-wrhv2" event={"ID":"f53b8e84-9062-4265-9b5e-4d4b40ad7020","Type":"ContainerStarted","Data":"9577eadd279713e67ba4d1eb85aacaaef1bc4d978457b80879c8aad61e8a6a47"}
Sep 15 22:28:16.336617 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.336120    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:28:16.336617 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.336202    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f" event={"ID":"b68f8430-b1b6-46d9-8358-74f04b66c9d6","Type":"ContainerStarted","Data":"73b1f1eac93c90c0dd31bdf7350cf3146133fca1123b38be304ac411d20069ab"}
Sep 15 22:28:16.338384 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.337704    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:28:16.339997 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.339938    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sm96j" event={"ID":"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028","Type":"ContainerStarted","Data":"1f82313f86a78a94afda15af92cdd8c442c7de29a65eafa64c9a721526767551"}
Sep 15 22:28:16.342797 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.342332    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pc4cx" event={"ID":"e8d2b3b9-6751-4c72-9677-dabc86c07f2a","Type":"ContainerStarted","Data":"9c68ef646f01360eaee1894f0b16f14fc27ba266b3091c9518ef5fbf98d4aa8e"}
Sep 15 22:28:16.342797 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.342376    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pc4cx" event={"ID":"e8d2b3b9-6751-4c72-9677-dabc86c07f2a","Type":"ContainerStarted","Data":"eb0709f8a01ed1166a3bdc32c039f8f68f030318dd3ce87c930e305a85854f6e"}
Sep 15 22:28:16.345892 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.345834    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x" event={"ID":"7f812033-940b-4211-bd1c-2eceec145d80","Type":"ContainerStarted","Data":"d1655c1af7d03aab496d860d01bd11ec40ea5dd5008d108646c158093bd54f85"}
Sep 15 22:28:16.357289 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.356811    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6df89d9594-wrhv2" podStartSLOduration=68.356794578 podStartE2EDuration="1m8.356794578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:16.355204648 +0000 UTC m=+68.974019558" watchObservedRunningTime="2026-09-15 22:28:16.356794578 +0000 UTC m=+68.975609481"
Sep 15 22:28:16.377324 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.377258    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pc4cx" podStartSLOduration=68.108243234 podStartE2EDuration="1m9.377241247s" totalImagesPullingTime="1.268998013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:16.376608179 +0000 UTC m=+68.995423088" watchObservedRunningTime="2026-09-15 22:28:16.377241247 +0000 UTC m=+68.996056156"
Sep 15 22:28:16.396537 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.395306    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f" podStartSLOduration=48.611064821 podStartE2EDuration="58.395289171s" totalImagesPullingTime="9.78422435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:16.394660768 +0000 UTC m=+69.013475677" watchObservedRunningTime="2026-09-15 22:28:16.395289171 +0000 UTC m=+69.014104080"
Sep 15 22:28:16.905595 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.904241    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:28:16.907728 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:16.907702    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:28:17.327046 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:17.327014    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-9xx4f"
Sep 15 22:28:17.354286 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:17.353828    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:28:17.359050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:17.355397    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6df89d9594-wrhv2"
Sep 15 22:28:17.567101 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:17.567045    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:17.637326 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:17.637203    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:17.810103 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:17.810069    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:17.985654 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:17.985576    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:18.196999 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:18.196962    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:18.384008 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:18.383974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:18.452540 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:18.452504    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:19.390502 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:19.390469    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:19.490983 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:19.490935    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:19.597419 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:19.597380    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:19.761893 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:19.761809    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:20.238327 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.238216    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:20.339032 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.338953    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:20.373570 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.373479    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mn2ls" event={"ID":"7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c","Type":"ContainerStarted","Data":"413427466f1f242916189032fc88d90858b6452f93ce8230cb804be1fb6b4875"}
Sep 15 22:28:20.376559 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.376479    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sm96j" event={"ID":"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028","Type":"ContainerStarted","Data":"999c1b77107fe21ce6b98eb1c9f98e8003ca0d2faef953702eeaecbfa088659a"}
Sep 15 22:28:20.376559 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.376526    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sm96j" event={"ID":"0eea82c0-60f7-4ee7-9a1d-8bd13f06c028","Type":"ContainerStarted","Data":"30b247aac60eedec05090acd0032b0dd54fe1181866287576153ab1a35c06800"}
Sep 15 22:28:20.376845 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.376790    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sm96j"
Sep 15 22:28:20.378575 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.378552    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h" event={"ID":"67b271e9-a2b8-4f33-be0b-102e434588ac","Type":"ContainerStarted","Data":"7303825071e5b633b9b374b4bce13e7d58f4292d8b8c97f5d70401facf7c5d60"}
Sep 15 22:28:20.380540 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.380516    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x" event={"ID":"7f812033-940b-4211-bd1c-2eceec145d80","Type":"ContainerStarted","Data":"46698bee846f7f34688a5e27a7c70223857c5fdaa731c22fb64d77382a0f53c1"}
Sep 15 22:28:20.380622 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.380563    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x" event={"ID":"7f812033-940b-4211-bd1c-2eceec145d80","Type":"ContainerStarted","Data":"49dda136689c8759330b0ef4105d2ede083fcb2c2094d037c5a178e54cea3909"}
Sep 15 22:28:20.392729 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.392679    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mn2ls" podStartSLOduration=66.405154502 podStartE2EDuration="1m12.392661891s" totalImagesPullingTime="5.987507389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:20.391253307 +0000 UTC m=+73.010068219" watchObservedRunningTime="2026-09-15 22:28:20.392661891 +0000 UTC m=+73.011476803"
Sep 15 22:28:20.427488 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.427437    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-96r9x" podStartSLOduration=58.066178073 podStartE2EDuration="1m2.427421677s" totalImagesPullingTime="4.361243604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:20.408942226 +0000 UTC m=+73.027757132" watchObservedRunningTime="2026-09-15 22:28:20.427421677 +0000 UTC m=+73.046236576"
Sep 15 22:28:20.427954 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.427924    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-gx49h" podStartSLOduration=67.72669865 podStartE2EDuration="1m12.427914906s" totalImagesPullingTime="4.701216256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:20.426924858 +0000 UTC m=+73.045739767" watchObservedRunningTime="2026-09-15 22:28:20.427914906 +0000 UTC m=+73.046729818"
Sep 15 22:28:20.435784 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.435760    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:20.446656 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.446616    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sm96j" podStartSLOduration=34.017195652 podStartE2EDuration="38.446601076s" totalImagesPullingTime="4.429405424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:20.445567813 +0000 UTC m=+73.064382790" watchObservedRunningTime="2026-09-15 22:28:20.446601076 +0000 UTC m=+73.065415984"
Sep 15 22:28:20.602330 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:20.602295    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:22.599173 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.599143    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n"]
Sep 15 22:28:22.628011 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.627982    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"]
Sep 15 22:28:22.628149 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.628093    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n"
Sep 15 22:28:22.643142 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.643006    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-kcx6s\""
Sep 15 22:28:22.643142 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.643021    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 22:28:22.643142 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.643073    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 22:28:22.643390 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.643345    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 15 22:28:22.643390 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.643345    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 22:28:22.643933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.643910    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm"]
Sep 15 22:28:22.644025 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.644009    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"
Sep 15 22:28:22.653073 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.653054    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 15 22:28:22.662025 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.662006    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"]
Sep 15 22:28:22.662159 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.662142    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm"
Sep 15 22:28:22.665190 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.665172    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 15 22:28:22.665409 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.665395    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tmf4d\""
Sep 15 22:28:22.665483 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.665428    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 15 22:28:22.688261 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.688229    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n"]
Sep 15 22:28:22.688435 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.688295    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm"]
Sep 15 22:28:22.688435 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.688326    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"]
Sep 15 22:28:22.688435 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.688364    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.689118 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.689093    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"]
Sep 15 22:28:22.691345 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.691321    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 15 22:28:22.691460 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.691354    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 15 22:28:22.692211 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.692194    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 15 22:28:22.692746 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.692732    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 15 22:28:22.734501 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.734476    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-6tfr7"]
Sep 15 22:28:22.737683 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.737656    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rksld\" (UniqueName: \"kubernetes.io/projected/f55a04d7-3868-466f-b014-d52726a36a1b-kube-api-access-rksld\") pod \"managed-serviceaccount-addon-agent-745894cc58-jks8n\" (UID: \"f55a04d7-3868-466f-b014-d52726a36a1b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n"
Sep 15 22:28:22.737777 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.737687    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f55a04d7-3868-466f-b014-d52726a36a1b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-745894cc58-jks8n\" (UID: \"f55a04d7-3868-466f-b014-d52726a36a1b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n"
Sep 15 22:28:22.737777 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.737710    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b022bc2b-4863-4452-a713-c41f862bf7ec-tmp\") pod \"klusterlet-addon-workmgr-558b4fbf49-lbkwh\" (UID: \"b022bc2b-4863-4452-a713-c41f862bf7ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"
Sep 15 22:28:22.737777 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.737733    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8r67\" (UniqueName: \"kubernetes.io/projected/b022bc2b-4863-4452-a713-c41f862bf7ec-kube-api-access-d8r67\") pod \"klusterlet-addon-workmgr-558b4fbf49-lbkwh\" (UID: \"b022bc2b-4863-4452-a713-c41f862bf7ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"
Sep 15 22:28:22.737875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.737785    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b022bc2b-4863-4452-a713-c41f862bf7ec-klusterlet-config\") pod \"klusterlet-addon-workmgr-558b4fbf49-lbkwh\" (UID: \"b022bc2b-4863-4452-a713-c41f862bf7ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"
Sep 15 22:28:22.758133 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.758110    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-6tfr7"
Sep 15 22:28:22.758865 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.758840    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:22.776893 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.776873    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 22:28:22.777164 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.777149    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 22:28:22.777417 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.777402    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zrkpn\""
Sep 15 22:28:22.804718 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.804693    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-6tfr7"]
Sep 15 22:28:22.838724 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.838702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksld\" (UniqueName: \"kubernetes.io/projected/f55a04d7-3868-466f-b014-d52726a36a1b-kube-api-access-rksld\") pod \"managed-serviceaccount-addon-agent-745894cc58-jks8n\" (UID: \"f55a04d7-3868-466f-b014-d52726a36a1b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n"
Sep 15 22:28:22.838851 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.838740    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-ca\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.838851 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.838758    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-hub\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.838851 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.838775    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f55a04d7-3868-466f-b014-d52726a36a1b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-745894cc58-jks8n\" (UID: \"f55a04d7-3868-466f-b014-d52726a36a1b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n"
Sep 15 22:28:22.838851 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.838794    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.838851 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.838810    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1f19bd34-abe0-4e3d-9245-6d577ed14679-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-gkrjm\" (UID: \"1f19bd34-abe0-4e3d-9245-6d577ed14679\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm"
Sep 15 22:28:22.839068 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.838920    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b022bc2b-4863-4452-a713-c41f862bf7ec-tmp\") pod \"klusterlet-addon-workmgr-558b4fbf49-lbkwh\" (UID: \"b022bc2b-4863-4452-a713-c41f862bf7ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"
Sep 15 22:28:22.839068 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.838949    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9pf\" (UniqueName: \"kubernetes.io/projected/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-kube-api-access-gv9pf\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.839068 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.838967    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8r67\" (UniqueName: \"kubernetes.io/projected/b022bc2b-4863-4452-a713-c41f862bf7ec-kube-api-access-d8r67\") pod \"klusterlet-addon-workmgr-558b4fbf49-lbkwh\" (UID: \"b022bc2b-4863-4452-a713-c41f862bf7ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"
Sep 15 22:28:22.839068 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.838984    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1f19bd34-abe0-4e3d-9245-6d577ed14679-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-gkrjm\" (UID: \"1f19bd34-abe0-4e3d-9245-6d577ed14679\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm"
Sep 15 22:28:22.839068 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.839031    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsbzf\" (UniqueName: \"kubernetes.io/projected/85c6151f-5846-4b92-b686-4a4efcbcc1ce-kube-api-access-gsbzf\") pod \"downloads-76f99584f4-6tfr7\" (UID: \"85c6151f-5846-4b92-b686-4a4efcbcc1ce\") " pod="openshift-console/downloads-76f99584f4-6tfr7"
Sep 15 22:28:22.839352 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.839066    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b022bc2b-4863-4452-a713-c41f862bf7ec-klusterlet-config\") pod \"klusterlet-addon-workmgr-558b4fbf49-lbkwh\" (UID: \"b022bc2b-4863-4452-a713-c41f862bf7ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"
Sep 15 22:28:22.839352 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.839112    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.839352 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.839148    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.839352 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.839343    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b022bc2b-4863-4452-a713-c41f862bf7ec-tmp\") pod \"klusterlet-addon-workmgr-558b4fbf49-lbkwh\" (UID: \"b022bc2b-4863-4452-a713-c41f862bf7ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"
Sep 15 22:28:22.841431 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.841415    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f55a04d7-3868-466f-b014-d52726a36a1b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-745894cc58-jks8n\" (UID: \"f55a04d7-3868-466f-b014-d52726a36a1b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n"
Sep 15 22:28:22.841543 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.841524    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b022bc2b-4863-4452-a713-c41f862bf7ec-klusterlet-config\") pod \"klusterlet-addon-workmgr-558b4fbf49-lbkwh\" (UID: \"b022bc2b-4863-4452-a713-c41f862bf7ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"
Sep 15 22:28:22.862837 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.862775    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cjtlt"]
Sep 15 22:28:22.876808 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.876792    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cjtlt"
Sep 15 22:28:22.887290 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.887255    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 15 22:28:22.887385 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.887370    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rksld\" (UniqueName: \"kubernetes.io/projected/f55a04d7-3868-466f-b014-d52726a36a1b-kube-api-access-rksld\") pod \"managed-serviceaccount-addon-agent-745894cc58-jks8n\" (UID: \"f55a04d7-3868-466f-b014-d52726a36a1b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n"
Sep 15 22:28:22.887903 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.887887    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-p7kxn\""
Sep 15 22:28:22.894200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.894183    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8r67\" (UniqueName: \"kubernetes.io/projected/b022bc2b-4863-4452-a713-c41f862bf7ec-kube-api-access-d8r67\") pod \"klusterlet-addon-workmgr-558b4fbf49-lbkwh\" (UID: \"b022bc2b-4863-4452-a713-c41f862bf7ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"
Sep 15 22:28:22.917550 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.917527    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cjtlt"]
Sep 15 22:28:22.940279 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.940243    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-ca\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.940380 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.940280    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-hub\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.940380 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.940301    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.940380 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.940317    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1f19bd34-abe0-4e3d-9245-6d577ed14679-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-gkrjm\" (UID: \"1f19bd34-abe0-4e3d-9245-6d577ed14679\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm"
Sep 15 22:28:22.940380 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.940352    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9pf\" (UniqueName: \"kubernetes.io/projected/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-kube-api-access-gv9pf\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.940380 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.940378    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1f19bd34-abe0-4e3d-9245-6d577ed14679-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-gkrjm\" (UID: \"1f19bd34-abe0-4e3d-9245-6d577ed14679\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm"
Sep 15 22:28:22.940611 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.940402    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsbzf\" (UniqueName: \"kubernetes.io/projected/85c6151f-5846-4b92-b686-4a4efcbcc1ce-kube-api-access-gsbzf\") pod \"downloads-76f99584f4-6tfr7\" (UID: \"85c6151f-5846-4b92-b686-4a4efcbcc1ce\") " pod="openshift-console/downloads-76f99584f4-6tfr7"
Sep 15 22:28:22.940611 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.940436    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.940611 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.940463    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.941359 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.941325    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.941359 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.941343    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1f19bd34-abe0-4e3d-9245-6d577ed14679-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-gkrjm\" (UID: \"1f19bd34-abe0-4e3d-9245-6d577ed14679\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm"
Sep 15 22:28:22.943120 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.943096    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1f19bd34-abe0-4e3d-9245-6d577ed14679-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-gkrjm\" (UID: \"1f19bd34-abe0-4e3d-9245-6d577ed14679\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm"
Sep 15 22:28:22.943286 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.943243    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-ca\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.943419 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.943404    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.943472 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.943450    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-hub\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.943527 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.943498    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.947221 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.947195    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n"
Sep 15 22:28:22.955247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.955228    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"
Sep 15 22:28:22.968536 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.968519    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsbzf\" (UniqueName: \"kubernetes.io/projected/85c6151f-5846-4b92-b686-4a4efcbcc1ce-kube-api-access-gsbzf\") pod \"downloads-76f99584f4-6tfr7\" (UID: \"85c6151f-5846-4b92-b686-4a4efcbcc1ce\") " pod="openshift-console/downloads-76f99584f4-6tfr7"
Sep 15 22:28:22.970259 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.970242    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm"
Sep 15 22:28:22.981061 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.981044    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9pf\" (UniqueName: \"kubernetes.io/projected/00d560c0-86f8-4b8f-94ab-7f0a9e3a682e-kube-api-access-gv9pf\") pod \"cluster-proxy-proxy-agent-76f8776769-p84kw\" (UID: \"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:22.996961 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:22.996939    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"
Sep 15 22:28:23.041317 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.041282    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3b2a046f-4acd-42ed-825e-3e5b8a69ebc5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cjtlt\" (UID: \"3b2a046f-4acd-42ed-825e-3e5b8a69ebc5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cjtlt"
Sep 15 22:28:23.063863 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.063816    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:23.082962 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.082797    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-6tfr7"
Sep 15 22:28:23.142234 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.142164    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3b2a046f-4acd-42ed-825e-3e5b8a69ebc5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cjtlt\" (UID: \"3b2a046f-4acd-42ed-825e-3e5b8a69ebc5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cjtlt"
Sep 15 22:28:23.146493 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.146144    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3b2a046f-4acd-42ed-825e-3e5b8a69ebc5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cjtlt\" (UID: \"3b2a046f-4acd-42ed-825e-3e5b8a69ebc5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cjtlt"
Sep 15 22:28:23.185557 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.185534    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cjtlt"
Sep 15 22:28:23.267948 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.267920    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n"]
Sep 15 22:28:23.270590 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.270563    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:23.272881 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:28:23.272853    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf55a04d7_3868_466f_b014_d52726a36a1b.slice/crio-8499d3ea787e504b2fb5c5466c7ba1ee30f2323bd80bc3d5f6cfb68abfc204c1 WatchSource:0}: Error finding container 8499d3ea787e504b2fb5c5466c7ba1ee30f2323bd80bc3d5f6cfb68abfc204c1: Status 404 returned error can't find the container with id 8499d3ea787e504b2fb5c5466c7ba1ee30f2323bd80bc3d5f6cfb68abfc204c1
Sep 15 22:28:23.325408 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.325384    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"]
Sep 15 22:28:23.327093 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:28:23.327072    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb022bc2b_4863_4452_a713_c41f862bf7ec.slice/crio-ed78a62439660ddb95b8be833fe47954aa3f30df6cb0d23bd05acfa91675b5b0 WatchSource:0}: Error finding container ed78a62439660ddb95b8be833fe47954aa3f30df6cb0d23bd05acfa91675b5b0: Status 404 returned error can't find the container with id ed78a62439660ddb95b8be833fe47954aa3f30df6cb0d23bd05acfa91675b5b0
Sep 15 22:28:23.358505 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.358480    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw"]
Sep 15 22:28:23.360185 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:28:23.360164    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00d560c0_86f8_4b8f_94ab_7f0a9e3a682e.slice/crio-62839578a848cc41cc748562cad18a9d644b95bc084f2d7d7bc140d8c0969fb2 WatchSource:0}: Error finding container 62839578a848cc41cc748562cad18a9d644b95bc084f2d7d7bc140d8c0969fb2: Status 404 returned error can't find the container with id 62839578a848cc41cc748562cad18a9d644b95bc084f2d7d7bc140d8c0969fb2
Sep 15 22:28:23.393563 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.393439    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm"]
Sep 15 22:28:23.395322 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.395293    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw" event={"ID":"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e","Type":"ContainerStarted","Data":"62839578a848cc41cc748562cad18a9d644b95bc084f2d7d7bc140d8c0969fb2"}
Sep 15 22:28:23.395728 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:28:23.395704    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f19bd34_abe0_4e3d_9245_6d577ed14679.slice/crio-23eafcc4fed297bbd67fe566862cdb723648428dacb3177bb8cef94aabfd3840 WatchSource:0}: Error finding container 23eafcc4fed297bbd67fe566862cdb723648428dacb3177bb8cef94aabfd3840: Status 404 returned error can't find the container with id 23eafcc4fed297bbd67fe566862cdb723648428dacb3177bb8cef94aabfd3840
Sep 15 22:28:23.396638 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.396614    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n" event={"ID":"f55a04d7-3868-466f-b014-d52726a36a1b","Type":"ContainerStarted","Data":"8499d3ea787e504b2fb5c5466c7ba1ee30f2323bd80bc3d5f6cfb68abfc204c1"}
Sep 15 22:28:23.397846 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.397827    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh" event={"ID":"b022bc2b-4863-4452-a713-c41f862bf7ec","Type":"ContainerStarted","Data":"ed78a62439660ddb95b8be833fe47954aa3f30df6cb0d23bd05acfa91675b5b0"}
Sep 15 22:28:23.417244 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.417226    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-6tfr7"]
Sep 15 22:28:23.419289 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:28:23.419244    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c6151f_5846_4b92_b686_4a4efcbcc1ce.slice/crio-6638c746599c23a7f5fa117e74cbaf76c1d9b3ac5e36d4d4c8108a74715e6946 WatchSource:0}: Error finding container 6638c746599c23a7f5fa117e74cbaf76c1d9b3ac5e36d4d4c8108a74715e6946: Status 404 returned error can't find the container with id 6638c746599c23a7f5fa117e74cbaf76c1d9b3ac5e36d4d4c8108a74715e6946
Sep 15 22:28:23.437619 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.437598    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:23.495426 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.495404    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cjtlt"]
Sep 15 22:28:23.497534 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:28:23.497505    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2a046f_4acd_42ed_825e_3e5b8a69ebc5.slice/crio-064f47609d88a2fbe3ed1ce51d8a7bdc052f08bc47cc367f5571f299c4e06c11 WatchSource:0}: Error finding container 064f47609d88a2fbe3ed1ce51d8a7bdc052f08bc47cc367f5571f299c4e06c11: Status 404 returned error can't find the container with id 064f47609d88a2fbe3ed1ce51d8a7bdc052f08bc47cc367f5571f299c4e06c11
Sep 15 22:28:23.857138 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.857092    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:23.976694 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:23.976655    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:24.409077 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:24.408980    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cjtlt" event={"ID":"3b2a046f-4acd-42ed-825e-3e5b8a69ebc5","Type":"ContainerStarted","Data":"064f47609d88a2fbe3ed1ce51d8a7bdc052f08bc47cc367f5571f299c4e06c11"}
Sep 15 22:28:24.412127 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:24.412025    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm" event={"ID":"1f19bd34-abe0-4e3d-9245-6d577ed14679","Type":"ContainerStarted","Data":"23eafcc4fed297bbd67fe566862cdb723648428dacb3177bb8cef94aabfd3840"}
Sep 15 22:28:24.415902 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:24.415853    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-6tfr7" event={"ID":"85c6151f-5846-4b92-b686-4a4efcbcc1ce","Type":"ContainerStarted","Data":"6638c746599c23a7f5fa117e74cbaf76c1d9b3ac5e36d4d4c8108a74715e6946"}
Sep 15 22:28:28.714638 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:28.714597    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:28.851063 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:28.851015    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:28.927506 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:28.927468    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:30.388759 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:30.388630    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sm96j"
Sep 15 22:28:30.939441 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:30.939404    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:31.037893 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:31.037856    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:31.831416 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:31.831314    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:32.454887 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.454791    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:32.471242 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.471199    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw" event={"ID":"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e","Type":"ContainerStarted","Data":"051371a9d4cd6f6a5a384dc304ec6b4bb902ef04ddf12fa309c5f83b4f4b1213"}
Sep 15 22:28:32.471428 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.471255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw" event={"ID":"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e","Type":"ContainerStarted","Data":"286b056ff500c7c2a13fa753ac9922e4b65bbb5d56b46eedcf15cfe4a3c69d7e"}
Sep 15 22:28:32.471428 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.471299    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw" event={"ID":"00d560c0-86f8-4b8f-94ab-7f0a9e3a682e","Type":"ContainerStarted","Data":"7909db4d44f54ba7a227411979f4e8efcd662ba44f2fc40e92693f23cf67ae86"}
Sep 15 22:28:32.473042 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.473013    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n" event={"ID":"f55a04d7-3868-466f-b014-d52726a36a1b","Type":"ContainerStarted","Data":"36e1fd9909088bcab1af21539a3541e0e2ab5e3d444b7564cbde96c3489a2862"}
Sep 15 22:28:32.474767 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.474695    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh" event={"ID":"b022bc2b-4863-4452-a713-c41f862bf7ec","Type":"ContainerStarted","Data":"37ff85f6a10313f580724b3e564304ff01e389b39f363fb03c2bbbbee452c0bf"}
Sep 15 22:28:32.474885 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.474773    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"
Sep 15 22:28:32.476601 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.476572    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cjtlt" event={"ID":"3b2a046f-4acd-42ed-825e-3e5b8a69ebc5","Type":"ContainerStarted","Data":"7af3d548204aee9bf44685529130f52d92eeb6a64f917fd396020aa8dbbf8116"}
Sep 15 22:28:32.476741 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.476720    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh"
Sep 15 22:28:32.476812 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.476764    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cjtlt"
Sep 15 22:28:32.479104 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.479075    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm" event={"ID":"1f19bd34-abe0-4e3d-9245-6d577ed14679","Type":"ContainerStarted","Data":"4e9c106bd5de568ca2be76cfbd220c30089117c0bc7e4776b97143fd3121428c"}
Sep 15 22:28:32.481924 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.481907    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cjtlt"
Sep 15 22:28:32.501831 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.501772    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8776769-p84kw" podStartSLOduration=1.908448242 podStartE2EDuration="10.501758271s" totalImagesPullingTime="8.593310029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:32.499167555 +0000 UTC m=+85.117982466" watchObservedRunningTime="2026-09-15 22:28:32.501758271 +0000 UTC m=+85.120573237"
Sep 15 22:28:32.523285 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.523226    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gkrjm" podStartSLOduration=1.991982155 podStartE2EDuration="10.523206401s" totalImagesPullingTime="8.531224246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:32.522908775 +0000 UTC m=+85.141723683" watchObservedRunningTime="2026-09-15 22:28:32.523206401 +0000 UTC m=+85.142021310"
Sep 15 22:28:32.545509 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.545455    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cjtlt" podStartSLOduration=2.115972461 podStartE2EDuration="10.545433701s" totalImagesPullingTime="8.42946124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:32.544780294 +0000 UTC m=+85.163595201" watchObservedRunningTime="2026-09-15 22:28:32.545433701 +0000 UTC m=+85.164248609"
Sep 15 22:28:32.568412 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.568368    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:32.589633 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.589589    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b4fbf49-lbkwh" podStartSLOduration=1.9560659660000002 podStartE2EDuration="10.589568273s" totalImagesPullingTime="8.633502307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:32.572344211 +0000 UTC m=+85.191159124" watchObservedRunningTime="2026-09-15 22:28:32.589568273 +0000 UTC m=+85.208383183"
Sep 15 22:28:32.590104 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.590082    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-745894cc58-jks8n" podStartSLOduration=1.936291432 podStartE2EDuration="10.590075283s" totalImagesPullingTime="8.653783851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:32.588728813 +0000 UTC m=+85.207543725" watchObservedRunningTime="2026-09-15 22:28:32.590075283 +0000 UTC m=+85.208890191"
Sep 15 22:28:32.661868 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.661824    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:32.735219 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:32.735086    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:33.486997 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:33.486964    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:33.564396 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:33.564359    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:33.720194 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:33.720143    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:34.891933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:34.891900    2584 patch_prober.go:28] interesting pod/image-registry-687df9c7d4-t72hr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 22:28:34.892404 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:34.891957    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-687df9c7d4-t72hr" podUID="0fc17f4f-84da-4a36-a3bc-c874fb9296e8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 22:28:35.471461 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:35.471408    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:35.537246 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:35.537200    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:35.628565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:35.628528    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:35.994449 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:35.994414    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:37.359185 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.359157    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:28:37.640832 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.640750    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:37.742412 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.742371    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:37.858287 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.858240    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:37.928662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.928546    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cdr4b"]
Sep 15 22:28:37.931987 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.931949    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:37.935232 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.935165    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 22:28:37.935439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.935421    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 22:28:37.936260 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.936238    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-99wnf\""
Sep 15 22:28:37.936850 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.936827    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 22:28:37.937060 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.937039    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 22:28:37.966201 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.966167    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:37.966371 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.966239    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30be65c9-2696-450f-831a-f05c587edcfa-root\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:37.966371 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.966295    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-wtmp\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:37.966371 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.966321    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30be65c9-2696-450f-831a-f05c587edcfa-sys\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:37.966371 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.966343    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-textfile\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:37.966371 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.966365    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:37.966851 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.966411    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz5jr\" (UniqueName: \"kubernetes.io/projected/30be65c9-2696-450f-831a-f05c587edcfa-kube-api-access-rz5jr\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:37.966851 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.966454    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30be65c9-2696-450f-831a-f05c587edcfa-metrics-client-ca\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:37.966851 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.966487    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-tls\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:37.969580 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:37.969552    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:38.041208 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.041154    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:38.066941 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.066906    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30be65c9-2696-450f-831a-f05c587edcfa-root\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067099 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.066963    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-wtmp\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067099 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.066996    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30be65c9-2696-450f-831a-f05c587edcfa-sys\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067099 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.067023    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-textfile\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067099 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.067049    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067099 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.067081    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30be65c9-2696-450f-831a-f05c587edcfa-sys\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067099 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.067092    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz5jr\" (UniqueName: \"kubernetes.io/projected/30be65c9-2696-450f-831a-f05c587edcfa-kube-api-access-rz5jr\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067423 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.067144    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30be65c9-2696-450f-831a-f05c587edcfa-metrics-client-ca\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067423 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.067151    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-wtmp\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067423 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.067174    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-tls\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067423 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.066989    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30be65c9-2696-450f-831a-f05c587edcfa-root\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067423 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.067207    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067423 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.067411    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-textfile\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067713 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:28:38.067574    2584 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 15 22:28:38.067713 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:28:38.067643    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-tls podName:30be65c9-2696-450f-831a-f05c587edcfa nodeName:}" failed. No retries permitted until 2026-09-15 22:28:38.567622465 +0000 UTC m=+91.186437357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-tls") pod "node-exporter-cdr4b" (UID: "30be65c9-2696-450f-831a-f05c587edcfa") : secret "node-exporter-tls" not found
Sep 15 22:28:38.067845 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.067824    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.067999 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.067942    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30be65c9-2696-450f-831a-f05c587edcfa-metrics-client-ca\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.070547 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.070503    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.077551 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.077507    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz5jr\" (UniqueName: \"kubernetes.io/projected/30be65c9-2696-450f-831a-f05c587edcfa-kube-api-access-rz5jr\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.129689 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.129646    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:38.230451 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.230363    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:38.335056 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.335018    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:38.434895 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.434850    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:38.570542 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.570506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-tls\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.574031 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.574004    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30be65c9-2696-450f-831a-f05c587edcfa-node-exporter-tls\") pod \"node-exporter-cdr4b\" (UID: \"30be65c9-2696-450f-831a-f05c587edcfa\") " pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.845979 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.845891    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cdr4b"
Sep 15 22:28:38.864353 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.864292    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:38.971309 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:38.971255    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:39.075831 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:39.075713    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:39.178351 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:39.178205    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:39.531423 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:39.531201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:39.693507 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:39.693469    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:40.094243 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:40.094204    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:40.189702 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:40.189661    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:40.290568 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:40.290532    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:40.833211 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:40.833147    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:40.894996 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:40.894957    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:41.103430 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:41.103313    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:41.667814 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:41.667776    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:42.202731 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:28:42.202699    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30be65c9_2696_450f_831a_f05c587edcfa.slice/crio-760b4fa367242682629ab3deff833b20d50d9ea75579bbe41dc230137cbde321 WatchSource:0}: Error finding container 760b4fa367242682629ab3deff833b20d50d9ea75579bbe41dc230137cbde321: Status 404 returned error can't find the container with id 760b4fa367242682629ab3deff833b20d50d9ea75579bbe41dc230137cbde321
Sep 15 22:28:42.227205 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:42.227181    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:42.324666 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:42.324627    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:42.521360 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:42.521292    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cdr4b" event={"ID":"30be65c9-2696-450f-831a-f05c587edcfa","Type":"ContainerStarted","Data":"760b4fa367242682629ab3deff833b20d50d9ea75579bbe41dc230137cbde321"}
Sep 15 22:28:42.666641 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:42.666555    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:42.779472 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:42.779390    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:43.218340 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:43.218300    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:43.408846 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:43.408805    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:43.529142 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:43.529053    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-6tfr7" event={"ID":"85c6151f-5846-4b92-b686-4a4efcbcc1ce","Type":"ContainerStarted","Data":"e771101a7bd5a15453c69be342599ebbf2ca12ec9d5ca10b2d8bbefdf0ef29d7"}
Sep 15 22:28:43.529679 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:43.529621    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-6tfr7"
Sep 15 22:28:43.530591 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:43.530560    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:43.541572 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:43.541549    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-6tfr7"
Sep 15 22:28:43.552056 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:43.552010    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-6tfr7" podStartSLOduration=2.40403899 podStartE2EDuration="21.551996251s" totalImagesPullingTime="19.147957261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:43.551331088 +0000 UTC m=+96.170145998" watchObservedRunningTime="2026-09-15 22:28:43.551996251 +0000 UTC m=+96.170811159"
Sep 15 22:28:44.107901 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:44.107804    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:44.537642 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:44.537558    2584 generic.go:356] "Generic (PLEG): container finished" podID="30be65c9-2696-450f-831a-f05c587edcfa" containerID="9e6c860d9d4eaffdc91ea083512c7bbc1906434bdfbd5d273a00bc2670de0a4e" exitCode=0
Sep 15 22:28:44.538070 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:44.537653    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cdr4b" event={"ID":"30be65c9-2696-450f-831a-f05c587edcfa","Type":"ContainerDied","Data":"9e6c860d9d4eaffdc91ea083512c7bbc1906434bdfbd5d273a00bc2670de0a4e"}
Sep 15 22:28:45.037598 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:45.037562    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:45.110450 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:45.110363    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:45.238136 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:45.238096    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:45.321352 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:45.321317    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jc5qb"
Sep 15 22:28:45.339713 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:45.339679    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:45.545174 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:45.545140    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cdr4b" event={"ID":"30be65c9-2696-450f-831a-f05c587edcfa","Type":"ContainerStarted","Data":"934f2b1b26c0317cce8d7644dec8144bd2026ad250054e265d0a8f1eb020ef5b"}
Sep 15 22:28:45.545677 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:45.545197    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cdr4b" event={"ID":"30be65c9-2696-450f-831a-f05c587edcfa","Type":"ContainerStarted","Data":"64237ed95dc774f2ba1a5fe518cf52c6030881f2ef18b38de609ea448459d345"}
Sep 15 22:28:45.566386 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:45.566329    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cdr4b" podStartSLOduration=7.063734883 podStartE2EDuration="8.56630934s" totalImagesPullingTime="1.40354307s" totalInitContainerRuntime="99.031387ms" isStatefulPod=true podCreationTimestamp="2026-09-15 22:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:28:45.564529234 +0000 UTC m=+98.183344157" watchObservedRunningTime="2026-09-15 22:28:45.56630934 +0000 UTC m=+98.185124249"
Sep 15 22:28:47.260652 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:47.260612    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:47.372833 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:47.372669    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:47.920458 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:47.920424    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-687df9c7d4-t72hr"]
Sep 15 22:28:48.039916 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:48.039859    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:48.161477 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:48.161437    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:49.315824 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:49.315783    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:49.415040 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:49.414998    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:49.515280 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:49.515229    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:52.522968 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:52.522940    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:52.628667 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:52.628634    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:55.326404 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:55.326358    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:55.382041 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:55.382009    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:57.537851 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:57.537812    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:57.602429 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:57.602392    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:28:59.608277 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:59.608243    2584 generic.go:356] "Generic (PLEG): container finished" podID="fc2ba350-9c05-4b47-9f16-8b19093b9137" containerID="6f089477fd7f2af3b60c0078c7590880c14e7ff41ec240a0a6d733725e9891a0" exitCode=0
Sep 15 22:28:59.608634 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:59.608305    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9x825" event={"ID":"fc2ba350-9c05-4b47-9f16-8b19093b9137","Type":"ContainerDied","Data":"6f089477fd7f2af3b60c0078c7590880c14e7ff41ec240a0a6d733725e9891a0"}
Sep 15 22:28:59.608679 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:28:59.608633    2584 scope.go:117] "RemoveContainer" containerID="6f089477fd7f2af3b60c0078c7590880c14e7ff41ec240a0a6d733725e9891a0"
Sep 15 22:29:00.613832 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:00.613799    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9x825" event={"ID":"fc2ba350-9c05-4b47-9f16-8b19093b9137","Type":"ContainerStarted","Data":"cf19249316c99cb2465b3a86045c1e6bfa09bc3f2e9991900094ec9723317ab7"}
Sep 15 22:29:04.378153 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:04.378116    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:29:04.438327 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:04.438292    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:29:09.649928 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:09.649891    2584 generic.go:356] "Generic (PLEG): container finished" podID="702fee44-5a8b-4364-a00f-957fab8661dd" containerID="43a236cfacf835efc9084548ac6551caa5abc151d756c14ec4ffed4a6aae039d" exitCode=0
Sep 15 22:29:09.650383 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:09.649946    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9" event={"ID":"702fee44-5a8b-4364-a00f-957fab8661dd","Type":"ContainerDied","Data":"43a236cfacf835efc9084548ac6551caa5abc151d756c14ec4ffed4a6aae039d"}
Sep 15 22:29:09.650383 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:09.650252    2584 scope.go:117] "RemoveContainer" containerID="43a236cfacf835efc9084548ac6551caa5abc151d756c14ec4ffed4a6aae039d"
Sep 15 22:29:10.655576 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:10.655544    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qc5w9" event={"ID":"702fee44-5a8b-4364-a00f-957fab8661dd","Type":"ContainerStarted","Data":"617e930b5d71f14ed765e6e79eee298a370d5ce4baed478d776fd77799e00eed"}
Sep 15 22:29:12.944466 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:12.944416    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-687df9c7d4-t72hr" podUID="0fc17f4f-84da-4a36-a3bc-c874fb9296e8" containerName="registry" containerID="cri-o://a4531a5bd6a1f4215c5334d0d743bc2cc4f543b146f9c193ef276c77d796d93a" gracePeriod=30
Sep 15 22:29:13.236830 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.236479    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:29:13.356257 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.356225    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-certificates\") pod \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") "
Sep 15 22:29:13.356443 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.356289    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-image-registry-private-configuration\") pod \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") "
Sep 15 22:29:13.356443 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.356318    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-bound-sa-token\") pod \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") "
Sep 15 22:29:13.356443 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.356342    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-trusted-ca\") pod \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") "
Sep 15 22:29:13.356443 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.356365    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxrjt\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-kube-api-access-qxrjt\") pod \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") "
Sep 15 22:29:13.356443 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.356406    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-ca-trust-extracted\") pod \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") "
Sep 15 22:29:13.356443 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.356435    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-installation-pull-secrets\") pod \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") "
Sep 15 22:29:13.356767 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.356465    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls\") pod \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\" (UID: \"0fc17f4f-84da-4a36-a3bc-c874fb9296e8\") "
Sep 15 22:29:13.356767 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.356658    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0fc17f4f-84da-4a36-a3bc-c874fb9296e8" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:29:13.356876 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.356822    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0fc17f4f-84da-4a36-a3bc-c874fb9296e8" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:29:13.359386 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.359290    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0fc17f4f-84da-4a36-a3bc-c874fb9296e8" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:29:13.359386 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.359333    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-kube-api-access-qxrjt" (OuterVolumeSpecName: "kube-api-access-qxrjt") pod "0fc17f4f-84da-4a36-a3bc-c874fb9296e8" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8"). InnerVolumeSpecName "kube-api-access-qxrjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:29:13.359561 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.359546    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0fc17f4f-84da-4a36-a3bc-c874fb9296e8" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:29:13.359713 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.359687    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0fc17f4f-84da-4a36-a3bc-c874fb9296e8" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:29:13.359818 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.359800    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0fc17f4f-84da-4a36-a3bc-c874fb9296e8" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:29:13.365605 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.365579    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0fc17f4f-84da-4a36-a3bc-c874fb9296e8" (UID: "0fc17f4f-84da-4a36-a3bc-c874fb9296e8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:29:13.457574 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.457484    2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-bound-sa-token\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:29:13.457574 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.457530    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-trusted-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:29:13.457574 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.457546    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxrjt\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-kube-api-access-qxrjt\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:29:13.457574 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.457565    2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-ca-trust-extracted\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:29:13.457804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.457582    2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-installation-pull-secrets\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:29:13.457804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.457599    2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-tls\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:29:13.457804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.457615    2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-registry-certificates\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:29:13.457804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.457633    2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0fc17f4f-84da-4a36-a3bc-c874fb9296e8-image-registry-private-configuration\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:29:13.674479 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.671826    2584 generic.go:356] "Generic (PLEG): container finished" podID="0fc17f4f-84da-4a36-a3bc-c874fb9296e8" containerID="a4531a5bd6a1f4215c5334d0d743bc2cc4f543b146f9c193ef276c77d796d93a" exitCode=0
Sep 15 22:29:13.674479 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.671925    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687df9c7d4-t72hr" event={"ID":"0fc17f4f-84da-4a36-a3bc-c874fb9296e8","Type":"ContainerDied","Data":"a4531a5bd6a1f4215c5334d0d743bc2cc4f543b146f9c193ef276c77d796d93a"}
Sep 15 22:29:13.674479 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.671985    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687df9c7d4-t72hr" event={"ID":"0fc17f4f-84da-4a36-a3bc-c874fb9296e8","Type":"ContainerDied","Data":"a571bc26c9c572fe0710b8ac3da5ede6eb8ece02fe3a71bad57e789a329ff252"}
Sep 15 22:29:13.674479 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.672017    2584 scope.go:117] "RemoveContainer" containerID="a4531a5bd6a1f4215c5334d0d743bc2cc4f543b146f9c193ef276c77d796d93a"
Sep 15 22:29:13.674479 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.672214    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687df9c7d4-t72hr"
Sep 15 22:29:13.703932 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.703912    2584 scope.go:117] "RemoveContainer" containerID="a4531a5bd6a1f4215c5334d0d743bc2cc4f543b146f9c193ef276c77d796d93a"
Sep 15 22:29:13.704332 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:29:13.704298    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4531a5bd6a1f4215c5334d0d743bc2cc4f543b146f9c193ef276c77d796d93a\": container with ID starting with a4531a5bd6a1f4215c5334d0d743bc2cc4f543b146f9c193ef276c77d796d93a not found: ID does not exist" containerID="a4531a5bd6a1f4215c5334d0d743bc2cc4f543b146f9c193ef276c77d796d93a"
Sep 15 22:29:13.704410 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.704348    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4531a5bd6a1f4215c5334d0d743bc2cc4f543b146f9c193ef276c77d796d93a"} err="failed to get container status \"a4531a5bd6a1f4215c5334d0d743bc2cc4f543b146f9c193ef276c77d796d93a\": rpc error: code = NotFound desc = could not find container \"a4531a5bd6a1f4215c5334d0d743bc2cc4f543b146f9c193ef276c77d796d93a\": container with ID starting with a4531a5bd6a1f4215c5334d0d743bc2cc4f543b146f9c193ef276c77d796d93a not found: ID does not exist"
Sep 15 22:29:13.716216 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.716159    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-687df9c7d4-t72hr"]
Sep 15 22:29:13.720742 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.720717    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-687df9c7d4-t72hr"]
Sep 15 22:29:13.899049 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:13.899016    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fc17f4f-84da-4a36-a3bc-c874fb9296e8" path="/var/lib/kubelet/pods/0fc17f4f-84da-4a36-a3bc-c874fb9296e8/volumes"
Sep 15 22:29:19.031794 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:19.031760    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:29:24.716225 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:24.716190    2584 generic.go:356] "Generic (PLEG): container finished" podID="307013b9-cf57-4e9e-bddd-e9d51ef7c57b" containerID="babecfb52b4a8727dfeb76a9f092bf09c08c89c882fc1a2648cee6ed86064e0c" exitCode=0
Sep 15 22:29:24.716644 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:24.716286    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc" event={"ID":"307013b9-cf57-4e9e-bddd-e9d51ef7c57b","Type":"ContainerDied","Data":"babecfb52b4a8727dfeb76a9f092bf09c08c89c882fc1a2648cee6ed86064e0c"}
Sep 15 22:29:24.716644 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:24.716613    2584 scope.go:117] "RemoveContainer" containerID="babecfb52b4a8727dfeb76a9f092bf09c08c89c882fc1a2648cee6ed86064e0c"
Sep 15 22:29:25.722431 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:25.722392    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-c4qxc" event={"ID":"307013b9-cf57-4e9e-bddd-e9d51ef7c57b","Type":"ContainerStarted","Data":"80ae454a2aab50029c0c3ece930101a04bb24f1f08315687f34d521d138f1993"}
Sep 15 22:29:46.008584 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:46.008542    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:29:46.070450 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:46.070419    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:29:57.113750 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:57.113712    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:29:57.310395 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:29:57.310355    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:30:06.453542 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:30:06.453462    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:30:06.510044 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:30:06.510012    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:30:17.495072 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:30:17.495037    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:30:17.595114 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:30:17.595079    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:30:17.667982 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:30:17.667945    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:30:30.531535 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:30:30.531495    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:30:30.587115 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:30:30.587052    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:31:12.379409 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:31:12.379374    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:31:12.440446 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:31:12.440402    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:31:17.798128 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:31:17.798091    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:31:17.896655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:31:17.896608    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:31:24.351832 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:31:24.351799    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:31:24.408991 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:31:24.408958    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:31:28.737333 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:31:28.737297    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:31:28.838678 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:31:28.838642    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:31:28.905369 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:31:28.905335    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:31:34.393170 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:31:34.393059    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:31:34.493332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:31:34.493301    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:31:38.540341 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:31:38.540258    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:07.800568 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:07.800530    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:32:07.806685 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:07.806662    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:32:07.811382 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:07.811368    2584 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 22:32:17.379065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:17.379030    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:17.471658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:17.471625    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:39.646432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:39.646396    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:41.075063 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:41.075011    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:43.480735 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:43.480700    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:43.577532 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:43.577499    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:44.945852 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:44.945816    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:45.087480 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:45.087442    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:45.408911 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:45.408883    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:45.591091 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:45.591057    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:46.965077 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:46.965036    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:47.378553 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:47.378511    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:50.031780 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:50.031664    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:50.119512 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:50.119462    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:52.497200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:52.497164    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:52.596751 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:52.596716    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:32:55.713655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:32:55.713619    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:01.250690 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:01.250607    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:01.355054 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:01.355018    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:07.901781 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:07.901748    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:08.003839 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:08.003802    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:09.909461 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:09.909412    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:22.396592 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:22.396553    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:22.496212 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:22.496177    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:23.001011 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.000978    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:23.740888 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.740848    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:23.844181 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.844149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:23.898558 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.898527    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"]
Sep 15 22:33:23.898965 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.898945    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc17f4f-84da-4a36-a3bc-c874fb9296e8" containerName="registry"
Sep 15 22:33:23.898965 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.898967    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc17f4f-84da-4a36-a3bc-c874fb9296e8" containerName="registry"
Sep 15 22:33:23.899114 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.899031    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc17f4f-84da-4a36-a3bc-c874fb9296e8" containerName="registry"
Sep 15 22:33:23.900993 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.900979    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:23.903495 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.903474    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 22:33:23.903724 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.903706    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 22:33:23.903841 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.903822    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hh8gz\""
Sep 15 22:33:23.903905 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.903868    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 22:33:23.904088 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.904075    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 22:33:23.920586 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:23.920555    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"]
Sep 15 22:33:24.036402 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.036316    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-475rc\" (UniqueName: \"kubernetes.io/projected/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-kube-api-access-475rc\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.036402 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.036363    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-tmp\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.036402 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.036382    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-manifests\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.036402 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.036398    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-webhook-cert\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.036725 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.036494    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.097875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.097840    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:24.137504 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.137465    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-475rc\" (UniqueName: \"kubernetes.io/projected/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-kube-api-access-475rc\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.137671 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.137519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-tmp\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.137671 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.137536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-manifests\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.137671 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.137551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-webhook-cert\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.137671 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.137591    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.137877 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.137841    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-manifests\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.139647 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.139621    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-tmp\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.139935 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.139916    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-webhook-cert\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.140001 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.139947    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.145151 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.145133    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-475rc\" (UniqueName: \"kubernetes.io/projected/0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d-kube-api-access-475rc\") pod \"opendatahub-operator-controller-manager-749d6986f8-88k9d\" (UID: \"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d\") " pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.194142 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.194118    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:24.210516 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.210493    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:24.290780 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.290655    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:24.389132 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.389092    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:24.472003 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.471967    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:24.490478 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.490455    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"]
Sep 15 22:33:24.492216 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:33:24.492189    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a83cd6d_f5f3_4bd4_ae82_a1d6c06c288d.slice/crio-f6e0160299fbd0bafdf3f48150b9cbf7a2feb112300f375aabea23d22e9bb3e4 WatchSource:0}: Error finding container f6e0160299fbd0bafdf3f48150b9cbf7a2feb112300f375aabea23d22e9bb3e4: Status 404 returned error can't find the container with id f6e0160299fbd0bafdf3f48150b9cbf7a2feb112300f375aabea23d22e9bb3e4
Sep 15 22:33:24.493963 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.493945    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 22:33:24.549209 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.549187    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:24.647941 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.647912    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d" event={"ID":"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d","Type":"ContainerStarted","Data":"f6e0160299fbd0bafdf3f48150b9cbf7a2feb112300f375aabea23d22e9bb3e4"}
Sep 15 22:33:24.650213 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:24.650190    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:27.665699 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:27.665623    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d" event={"ID":"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d","Type":"ContainerStarted","Data":"949363827b3bc0921b56736a7940679333be47fdb9680ae98bd9cca201827a4e"}
Sep 15 22:33:28.194699 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:28.194646    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:28.321312 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:28.321277    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:28.671242 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:28.671205    2584 generic.go:356] "Generic (PLEG): container finished" podID="0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d" containerID="949363827b3bc0921b56736a7940679333be47fdb9680ae98bd9cca201827a4e" exitCode=0
Sep 15 22:33:28.671659 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:28.671296    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d" event={"ID":"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d","Type":"ContainerDied","Data":"949363827b3bc0921b56736a7940679333be47fdb9680ae98bd9cca201827a4e"}
Sep 15 22:33:29.678478 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:29.678438    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d" event={"ID":"0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d","Type":"ContainerStarted","Data":"35965efab86be185c4907b8344d887ac194a16d21d98f58bb44c37f68404b503"}
Sep 15 22:33:29.678848 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:29.678575    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:29.703485 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:29.703432    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d" podStartSLOduration=2.4036098040000002 podStartE2EDuration="6.703415302s" totalImagesPullingTime="3.280233224s" totalInitContainerRuntime="1.019572274s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:33:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:33:29.701195222 +0000 UTC m=+382.320010132" watchObservedRunningTime="2026-09-15 22:33:29.703415302 +0000 UTC m=+382.322230284"
Sep 15 22:33:33.978549 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:33.978516    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:34.078935 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:34.078890    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:39.076454 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:39.076424    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:39.217527 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:39.217490    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:40.685432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:40.685396    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-749d6986f8-88k9d"
Sep 15 22:33:42.947044 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:42.947010    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:43.084007 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:43.083966    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:45.943311 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:45.943258    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:46.265005 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:46.264911    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:47.230213 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:47.230172    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:47.295418 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:47.295385    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:48.274689 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:48.274653    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:50.424917 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:50.424882    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:50.485034 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:50.484999    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:50.564704 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:50.564667    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:52.119623 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:52.119576    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:52.228698 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:52.228664    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:52.998672 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:52.998638    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:53.154997 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:53.154957    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:53.257829 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:53.257728    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:54.134947 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:54.134911    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:54.233046 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:54.233014    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:54.333450 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:54.333411    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:54.632633 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:54.632574    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:33:54.959677 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:33:54.959585    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:34:03.105022 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.104986    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"]
Sep 15 22:34:03.108177 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.108154    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.112856 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.112831    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-sr5jj\""
Sep 15 22:34:03.112985 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.112831    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 22:34:03.113472 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.113450    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 22:34:03.113605 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.113474    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 22:34:03.113605 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.113491    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 22:34:03.113605 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.113456    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 22:34:03.145925 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.145894    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"]
Sep 15 22:34:03.152233 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.152203    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l67vj\" (UniqueName: \"kubernetes.io/projected/8694a0ff-df44-40a4-8b6e-5386c170a94f-kube-api-access-l67vj\") pod \"lws-controller-manager-7bd6c5dcdd-tzzhw\" (UID: \"8694a0ff-df44-40a4-8b6e-5386c170a94f\") " pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.152479 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.152460    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8694a0ff-df44-40a4-8b6e-5386c170a94f-manager-config\") pod \"lws-controller-manager-7bd6c5dcdd-tzzhw\" (UID: \"8694a0ff-df44-40a4-8b6e-5386c170a94f\") " pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.152615 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.152578    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8694a0ff-df44-40a4-8b6e-5386c170a94f-cert\") pod \"lws-controller-manager-7bd6c5dcdd-tzzhw\" (UID: \"8694a0ff-df44-40a4-8b6e-5386c170a94f\") " pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.152718 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.152629    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8694a0ff-df44-40a4-8b6e-5386c170a94f-metrics-cert\") pod \"lws-controller-manager-7bd6c5dcdd-tzzhw\" (UID: \"8694a0ff-df44-40a4-8b6e-5386c170a94f\") " pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.253704 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.253667    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8694a0ff-df44-40a4-8b6e-5386c170a94f-metrics-cert\") pod \"lws-controller-manager-7bd6c5dcdd-tzzhw\" (UID: \"8694a0ff-df44-40a4-8b6e-5386c170a94f\") " pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.253868 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.253736    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l67vj\" (UniqueName: \"kubernetes.io/projected/8694a0ff-df44-40a4-8b6e-5386c170a94f-kube-api-access-l67vj\") pod \"lws-controller-manager-7bd6c5dcdd-tzzhw\" (UID: \"8694a0ff-df44-40a4-8b6e-5386c170a94f\") " pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.253868 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.253764    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8694a0ff-df44-40a4-8b6e-5386c170a94f-manager-config\") pod \"lws-controller-manager-7bd6c5dcdd-tzzhw\" (UID: \"8694a0ff-df44-40a4-8b6e-5386c170a94f\") " pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.253868 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.253786    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8694a0ff-df44-40a4-8b6e-5386c170a94f-cert\") pod \"lws-controller-manager-7bd6c5dcdd-tzzhw\" (UID: \"8694a0ff-df44-40a4-8b6e-5386c170a94f\") " pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.254502 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.254475    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8694a0ff-df44-40a4-8b6e-5386c170a94f-manager-config\") pod \"lws-controller-manager-7bd6c5dcdd-tzzhw\" (UID: \"8694a0ff-df44-40a4-8b6e-5386c170a94f\") " pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.256125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.256101    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8694a0ff-df44-40a4-8b6e-5386c170a94f-cert\") pod \"lws-controller-manager-7bd6c5dcdd-tzzhw\" (UID: \"8694a0ff-df44-40a4-8b6e-5386c170a94f\") " pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.256200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.256155    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8694a0ff-df44-40a4-8b6e-5386c170a94f-metrics-cert\") pod \"lws-controller-manager-7bd6c5dcdd-tzzhw\" (UID: \"8694a0ff-df44-40a4-8b6e-5386c170a94f\") " pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.271838 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.271812    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l67vj\" (UniqueName: \"kubernetes.io/projected/8694a0ff-df44-40a4-8b6e-5386c170a94f-kube-api-access-l67vj\") pod \"lws-controller-manager-7bd6c5dcdd-tzzhw\" (UID: \"8694a0ff-df44-40a4-8b6e-5386c170a94f\") " pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.418213 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.418129    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:03.718014 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.717987    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"]
Sep 15 22:34:03.720210 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:34:03.720170    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8694a0ff_df44_40a4_8b6e_5386c170a94f.slice/crio-86688dabf60ad16732ba8b5b4f7f667721be770d79220c68538e12e0c304023d WatchSource:0}: Error finding container 86688dabf60ad16732ba8b5b4f7f667721be770d79220c68538e12e0c304023d: Status 404 returned error can't find the container with id 86688dabf60ad16732ba8b5b4f7f667721be770d79220c68538e12e0c304023d
Sep 15 22:34:03.839087 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.839055    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw" event={"ID":"8694a0ff-df44-40a4-8b6e-5386c170a94f","Type":"ContainerStarted","Data":"86688dabf60ad16732ba8b5b4f7f667721be770d79220c68538e12e0c304023d"}
Sep 15 22:34:03.922004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:03.921959    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:34:04.023891 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:04.023775    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:34:04.119297 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:04.119250    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:34:04.260868 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:04.260819    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:34:04.414671 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:04.414637    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:34:06.861380 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:06.861338    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw" event={"ID":"8694a0ff-df44-40a4-8b6e-5386c170a94f","Type":"ContainerStarted","Data":"ab829b8e681dddb7b0280bc41431d56ae6014c8aebb0b466bf098b59848ba721"}
Sep 15 22:34:06.861763 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:06.861426    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:06.893873 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:06.893823    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw" podStartSLOduration=1.2106284569999999 podStartE2EDuration="3.893808872s" totalImagesPullingTime="2.683180415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:34:06.891409119 +0000 UTC m=+419.510224028" watchObservedRunningTime="2026-09-15 22:34:06.893808872 +0000 UTC m=+419.512623780"
Sep 15 22:34:16.082728 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:16.082691    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:34:16.144439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:16.144407    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:34:17.868076 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:17.868043    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7bd6c5dcdd-tzzhw"
Sep 15 22:34:39.749729 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:39.749626    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:34:39.807145 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:39.807109    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:34:40.902288 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:40.902233    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:34:54.951492 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:54.951456    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:34:55.035985 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:34:55.035942    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:07.909282 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:07.909210    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:10.321692 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:10.321654    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:10.553195 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:10.553157    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:10.710903 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:10.710816    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:10.834575 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:10.834533    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:12.861833 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:12.861788    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:13.518503 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:13.518467    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:13.706669 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:13.706612    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:13.945432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:13.945393    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:14.053135 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:14.053103    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:14.323109 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:14.323060    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:14.423150 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:14.423113    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:17.751292 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:17.751247    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:17.851629 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:17.851591    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:18.656788 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:18.656755    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:18.762356 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:18.762315    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:19.387697 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:19.387663    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vhj44"]
Sep 15 22:35:19.389883 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:19.389858    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vhj44"
Sep 15 22:35:19.392813 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:19.392794    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6mmnt\""
Sep 15 22:35:19.392904 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:19.392810    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 15 22:35:19.392904 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:19.392836    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 22:35:19.393012 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:19.392990    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 22:35:19.405103 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:19.405078    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vhj44"]
Sep 15 22:35:19.448793 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:19.448759    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jlp8\" (UniqueName: \"kubernetes.io/projected/0788b7e2-df30-4fe8-9f0e-c08beded8818-kube-api-access-9jlp8\") pod \"dns-operator-controller-manager-7b84cd7c9c-vhj44\" (UID: \"0788b7e2-df30-4fe8-9f0e-c08beded8818\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vhj44"
Sep 15 22:35:19.549863 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:19.549823    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jlp8\" (UniqueName: \"kubernetes.io/projected/0788b7e2-df30-4fe8-9f0e-c08beded8818-kube-api-access-9jlp8\") pod \"dns-operator-controller-manager-7b84cd7c9c-vhj44\" (UID: \"0788b7e2-df30-4fe8-9f0e-c08beded8818\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vhj44"
Sep 15 22:35:19.559246 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:19.559215    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jlp8\" (UniqueName: \"kubernetes.io/projected/0788b7e2-df30-4fe8-9f0e-c08beded8818-kube-api-access-9jlp8\") pod \"dns-operator-controller-manager-7b84cd7c9c-vhj44\" (UID: \"0788b7e2-df30-4fe8-9f0e-c08beded8818\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vhj44"
Sep 15 22:35:19.700054 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:19.699952    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vhj44"
Sep 15 22:35:19.972925 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:19.972899    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vhj44"]
Sep 15 22:35:19.975467 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:35:19.975442    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0788b7e2_df30_4fe8_9f0e_c08beded8818.slice/crio-6bb56a896eae992865bfe0f88cbe183ed1e0cfef24f33cce22ec72c4c454d6ef WatchSource:0}: Error finding container 6bb56a896eae992865bfe0f88cbe183ed1e0cfef24f33cce22ec72c4c454d6ef: Status 404 returned error can't find the container with id 6bb56a896eae992865bfe0f88cbe183ed1e0cfef24f33cce22ec72c4c454d6ef
Sep 15 22:35:20.173277 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:20.173230    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vhj44" event={"ID":"0788b7e2-df30-4fe8-9f0e-c08beded8818","Type":"ContainerStarted","Data":"6bb56a896eae992865bfe0f88cbe183ed1e0cfef24f33cce22ec72c4c454d6ef"}
Sep 15 22:35:20.736672 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:20.736632    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:20.823151 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:20.823105    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:23.190873 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:23.190837    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vhj44" event={"ID":"0788b7e2-df30-4fe8-9f0e-c08beded8818","Type":"ContainerStarted","Data":"3e43a31c4ee06835c6dc4135ed6b57431aa3727d1e069ff66594d5c4ee152125"}
Sep 15 22:35:23.191369 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:23.190920    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vhj44"
Sep 15 22:35:23.219580 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:23.216584    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vhj44" podStartSLOduration=1.746969376 podStartE2EDuration="4.216566116s" totalImagesPullingTime="2.46959674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:35:23.21369974 +0000 UTC m=+495.832514670" watchObservedRunningTime="2026-09-15 22:35:23.216566116 +0000 UTC m=+495.835381024"
Sep 15 22:35:23.449370 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:23.449256    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:23.536041 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:23.536009    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:24.373150 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:24.373103    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:24.537536 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:24.537498    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:24.601900 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:24.601865    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:24.734700 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:24.734609    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:24.839027 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:24.838992    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:29.737048 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:29.737014    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:29.839903 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:29.839864    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:29.937190 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:29.937151    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:34.196742 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:34.196713    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vhj44"
Sep 15 22:35:36.199671 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.199636    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"]
Sep 15 22:35:36.208504 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.208470    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"
Sep 15 22:35:36.215549 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.215526    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d7f88\""
Sep 15 22:35:36.226682 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.226657    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"]
Sep 15 22:35:36.289643 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.289604    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b76248a5-d4bb-44e2-97b9-dde6f933b6b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4tf84\" (UID: \"b76248a5-d4bb-44e2-97b9-dde6f933b6b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"
Sep 15 22:35:36.289801 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.289673    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bldzb\" (UniqueName: \"kubernetes.io/projected/b76248a5-d4bb-44e2-97b9-dde6f933b6b7-kube-api-access-bldzb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4tf84\" (UID: \"b76248a5-d4bb-44e2-97b9-dde6f933b6b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"
Sep 15 22:35:36.390926 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.390891    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b76248a5-d4bb-44e2-97b9-dde6f933b6b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4tf84\" (UID: \"b76248a5-d4bb-44e2-97b9-dde6f933b6b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"
Sep 15 22:35:36.391085 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.390945    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bldzb\" (UniqueName: \"kubernetes.io/projected/b76248a5-d4bb-44e2-97b9-dde6f933b6b7-kube-api-access-bldzb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4tf84\" (UID: \"b76248a5-d4bb-44e2-97b9-dde6f933b6b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"
Sep 15 22:35:36.391284 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.391241    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b76248a5-d4bb-44e2-97b9-dde6f933b6b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4tf84\" (UID: \"b76248a5-d4bb-44e2-97b9-dde6f933b6b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"
Sep 15 22:35:36.399972 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.399947    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:36.404532 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.404512    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bldzb\" (UniqueName: \"kubernetes.io/projected/b76248a5-d4bb-44e2-97b9-dde6f933b6b7-kube-api-access-bldzb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4tf84\" (UID: \"b76248a5-d4bb-44e2-97b9-dde6f933b6b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"
Sep 15 22:35:36.519089 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.519024    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"
Sep 15 22:35:36.531939 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.531907    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:36.800300 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:36.800260    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"]
Sep 15 22:35:36.802637 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:35:36.802608    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76248a5_d4bb_44e2_97b9_dde6f933b6b7.slice/crio-787f014ee1986234fa5282f48aab32bf29949244afa941f2883454db2a7ae080 WatchSource:0}: Error finding container 787f014ee1986234fa5282f48aab32bf29949244afa941f2883454db2a7ae080: Status 404 returned error can't find the container with id 787f014ee1986234fa5282f48aab32bf29949244afa941f2883454db2a7ae080
Sep 15 22:35:37.250986 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:37.250904    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84" event={"ID":"b76248a5-d4bb-44e2-97b9-dde6f933b6b7","Type":"ContainerStarted","Data":"787f014ee1986234fa5282f48aab32bf29949244afa941f2883454db2a7ae080"}
Sep 15 22:35:40.614948 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:40.614914    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:40.716029 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:40.715995    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:41.275497 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:41.275457    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84" event={"ID":"b76248a5-d4bb-44e2-97b9-dde6f933b6b7","Type":"ContainerStarted","Data":"5c7cf8b37d8568f855d59f0782029c24ab342c4dbca02807dc7046da92a51e7b"}
Sep 15 22:35:41.275710 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:41.275571    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"
Sep 15 22:35:41.297651 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:41.297607    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84" podStartSLOduration=1.560209161 podStartE2EDuration="5.297591086s" totalImagesPullingTime="3.737381925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:35:41.295616841 +0000 UTC m=+513.914431752" watchObservedRunningTime="2026-09-15 22:35:41.297591086 +0000 UTC m=+513.916405991"
Sep 15 22:35:44.389791 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:44.389757    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:44.446876 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:44.446846    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:44.585658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:44.585625    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:44.686646 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:44.686571    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:48.897047 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:48.897012    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"]
Sep 15 22:35:48.897474 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:48.897301    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84" podUID="b76248a5-d4bb-44e2-97b9-dde6f933b6b7" containerName="manager" containerID="cri-o://5c7cf8b37d8568f855d59f0782029c24ab342c4dbca02807dc7046da92a51e7b" gracePeriod=10
Sep 15 22:35:48.899114 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:48.899088    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"
Sep 15 22:35:49.136725 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.136701    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"
Sep 15 22:35:49.300890 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.300865    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b76248a5-d4bb-44e2-97b9-dde6f933b6b7-extensions-socket-volume\") pod \"b76248a5-d4bb-44e2-97b9-dde6f933b6b7\" (UID: \"b76248a5-d4bb-44e2-97b9-dde6f933b6b7\") "
Sep 15 22:35:49.301047 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.300902    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bldzb\" (UniqueName: \"kubernetes.io/projected/b76248a5-d4bb-44e2-97b9-dde6f933b6b7-kube-api-access-bldzb\") pod \"b76248a5-d4bb-44e2-97b9-dde6f933b6b7\" (UID: \"b76248a5-d4bb-44e2-97b9-dde6f933b6b7\") "
Sep 15 22:35:49.301445 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.301417    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76248a5-d4bb-44e2-97b9-dde6f933b6b7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b76248a5-d4bb-44e2-97b9-dde6f933b6b7" (UID: "b76248a5-d4bb-44e2-97b9-dde6f933b6b7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:35:49.303052 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.303033    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76248a5-d4bb-44e2-97b9-dde6f933b6b7-kube-api-access-bldzb" (OuterVolumeSpecName: "kube-api-access-bldzb") pod "b76248a5-d4bb-44e2-97b9-dde6f933b6b7" (UID: "b76248a5-d4bb-44e2-97b9-dde6f933b6b7"). InnerVolumeSpecName "kube-api-access-bldzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:35:49.313985 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.313956    2584 generic.go:356] "Generic (PLEG): container finished" podID="b76248a5-d4bb-44e2-97b9-dde6f933b6b7" containerID="5c7cf8b37d8568f855d59f0782029c24ab342c4dbca02807dc7046da92a51e7b" exitCode=0
Sep 15 22:35:49.314079 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.313997    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"
Sep 15 22:35:49.314079 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.314033    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84" event={"ID":"b76248a5-d4bb-44e2-97b9-dde6f933b6b7","Type":"ContainerDied","Data":"5c7cf8b37d8568f855d59f0782029c24ab342c4dbca02807dc7046da92a51e7b"}
Sep 15 22:35:49.314151 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.314084    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84" event={"ID":"b76248a5-d4bb-44e2-97b9-dde6f933b6b7","Type":"ContainerDied","Data":"787f014ee1986234fa5282f48aab32bf29949244afa941f2883454db2a7ae080"}
Sep 15 22:35:49.314151 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.314107    2584 scope.go:117] "RemoveContainer" containerID="5c7cf8b37d8568f855d59f0782029c24ab342c4dbca02807dc7046da92a51e7b"
Sep 15 22:35:49.325647 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.325626    2584 scope.go:117] "RemoveContainer" containerID="5c7cf8b37d8568f855d59f0782029c24ab342c4dbca02807dc7046da92a51e7b"
Sep 15 22:35:49.325914 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:35:49.325892    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c7cf8b37d8568f855d59f0782029c24ab342c4dbca02807dc7046da92a51e7b\": container with ID starting with 5c7cf8b37d8568f855d59f0782029c24ab342c4dbca02807dc7046da92a51e7b not found: ID does not exist" containerID="5c7cf8b37d8568f855d59f0782029c24ab342c4dbca02807dc7046da92a51e7b"
Sep 15 22:35:49.325963 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.325921    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c7cf8b37d8568f855d59f0782029c24ab342c4dbca02807dc7046da92a51e7b"} err="failed to get container status \"5c7cf8b37d8568f855d59f0782029c24ab342c4dbca02807dc7046da92a51e7b\": rpc error: code = NotFound desc = could not find container \"5c7cf8b37d8568f855d59f0782029c24ab342c4dbca02807dc7046da92a51e7b\": container with ID starting with 5c7cf8b37d8568f855d59f0782029c24ab342c4dbca02807dc7046da92a51e7b not found: ID does not exist"
Sep 15 22:35:49.339817 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.339795    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"]
Sep 15 22:35:49.345851 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.345830    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4tf84"]
Sep 15 22:35:49.402382 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.402355    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b76248a5-d4bb-44e2-97b9-dde6f933b6b7-extensions-socket-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:35:49.402382 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.402379    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bldzb\" (UniqueName: \"kubernetes.io/projected/b76248a5-d4bb-44e2-97b9-dde6f933b6b7-kube-api-access-bldzb\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:35:49.898747 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:49.898678    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b76248a5-d4bb-44e2-97b9-dde6f933b6b7" path="/var/lib/kubelet/pods/b76248a5-d4bb-44e2-97b9-dde6f933b6b7/volumes"
Sep 15 22:35:50.002614 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:50.002580    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:50.100869 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:50.100837    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:50.306523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:50.306471    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:50.407684 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:50.407645    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:50.512972 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:50.512939    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:56.293147 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:56.293109    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:35:56.392744 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:35:56.392707    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:05.232543 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.232471    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"]
Sep 15 22:36:05.232964 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.232948    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76248a5-d4bb-44e2-97b9-dde6f933b6b7" containerName="manager"
Sep 15 22:36:05.233013 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.232967    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76248a5-d4bb-44e2-97b9-dde6f933b6b7" containerName="manager"
Sep 15 22:36:05.233057 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.233046    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76248a5-d4bb-44e2-97b9-dde6f933b6b7" containerName="manager"
Sep 15 22:36:05.236219 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.236199    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.239246 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.239215    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 22:36:05.239376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.239303    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mx5pn\""
Sep 15 22:36:05.248823 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.248799    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"]
Sep 15 22:36:05.262607 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.262584    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:05.319256 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.319229    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.319401 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.319295    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.319401 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.319357    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phsws\" (UniqueName: \"kubernetes.io/projected/847f9278-fbd9-44ff-b475-62af5997177f-kube-api-access-phsws\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.319401 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.319383    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847f9278-fbd9-44ff-b475-62af5997177f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.319531 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.319418    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847f9278-fbd9-44ff-b475-62af5997177f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.319531 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.319437    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847f9278-fbd9-44ff-b475-62af5997177f-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.319598 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.319528    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.319598 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.319561    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.319598 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.319588    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.420530 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.420503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.420701 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.420540    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.420701 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.420570    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.420701 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.420589    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.420701 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.420611    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phsws\" (UniqueName: \"kubernetes.io/projected/847f9278-fbd9-44ff-b475-62af5997177f-kube-api-access-phsws\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.420701 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.420630    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847f9278-fbd9-44ff-b475-62af5997177f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.420701 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.420657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847f9278-fbd9-44ff-b475-62af5997177f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.420701 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.420680    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847f9278-fbd9-44ff-b475-62af5997177f-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.421063 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.420709    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.421063 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.420952    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.421174 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.421152    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.421228 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.421211    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.421385 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.421362    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.421556 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.421537    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847f9278-fbd9-44ff-b475-62af5997177f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.423008 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.422985    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847f9278-fbd9-44ff-b475-62af5997177f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.423369 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.423353    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847f9278-fbd9-44ff-b475-62af5997177f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.431038 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.431018    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847f9278-fbd9-44ff-b475-62af5997177f-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.431187 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.431166    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phsws\" (UniqueName: \"kubernetes.io/projected/847f9278-fbd9-44ff-b475-62af5997177f-kube-api-access-phsws\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8b2xb\" (UID: \"847f9278-fbd9-44ff-b475-62af5997177f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.546576 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.546554    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:05.578995 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.578965    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:05.821782 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:05.821759    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"]
Sep 15 22:36:05.823869 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:36:05.823837    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847f9278_fbd9_44ff_b475_62af5997177f.slice/crio-7ba20921728562f57e077d70984617a7ffd0a82252909c5c9aa83bc80bdad9c1 WatchSource:0}: Error finding container 7ba20921728562f57e077d70984617a7ffd0a82252909c5c9aa83bc80bdad9c1: Status 404 returned error can't find the container with id 7ba20921728562f57e077d70984617a7ffd0a82252909c5c9aa83bc80bdad9c1
Sep 15 22:36:06.394133 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:06.394093    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb" event={"ID":"847f9278-fbd9-44ff-b475-62af5997177f","Type":"ContainerStarted","Data":"7ba20921728562f57e077d70984617a7ffd0a82252909c5c9aa83bc80bdad9c1"}
Sep 15 22:36:08.325122 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:08.325076    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:36:08.325427 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:08.325174    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:36:08.325427 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:08.325210    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:36:08.403961 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:08.403925    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb" event={"ID":"847f9278-fbd9-44ff-b475-62af5997177f","Type":"ContainerStarted","Data":"1544fe57835f43c48af7d26f767803eb3fd08f8416d5588be617b68ab1f829b4"}
Sep 15 22:36:08.425367 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:08.425284    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb" podStartSLOduration=0.926407876 podStartE2EDuration="3.425241989s" totalImagesPullingTime="2.498834113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:36:08.424325612 +0000 UTC m=+541.043140520" watchObservedRunningTime="2026-09-15 22:36:08.425241989 +0000 UTC m=+541.044056898"
Sep 15 22:36:08.547584 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:08.547558    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:09.405864 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.405821    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:09.524759 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.524720    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:09.552408 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.552380    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:09.586500 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.586457    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:09.814393 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.814358    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:36:09.817828 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.817806    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-ql74p"
Sep 15 22:36:09.820543 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.820499    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 22:36:09.820771 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.820753    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-j5m44\""
Sep 15 22:36:09.826610 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.826586    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:36:09.859208 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.859178    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krz54\" (UniqueName: \"kubernetes.io/projected/f5a40e7f-6cff-4a38-a3a0-ce490ff37330-kube-api-access-krz54\") pod \"limitador-limitador-84994f7ffc-ql74p\" (UID: \"f5a40e7f-6cff-4a38-a3a0-ce490ff37330\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ql74p"
Sep 15 22:36:09.859518 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.859302    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f5a40e7f-6cff-4a38-a3a0-ce490ff37330-config-file\") pod \"limitador-limitador-84994f7ffc-ql74p\" (UID: \"f5a40e7f-6cff-4a38-a3a0-ce490ff37330\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ql74p"
Sep 15 22:36:09.860664 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.860645    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:36:09.960336 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.960302    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f5a40e7f-6cff-4a38-a3a0-ce490ff37330-config-file\") pod \"limitador-limitador-84994f7ffc-ql74p\" (UID: \"f5a40e7f-6cff-4a38-a3a0-ce490ff37330\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ql74p"
Sep 15 22:36:09.960461 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.960371    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krz54\" (UniqueName: \"kubernetes.io/projected/f5a40e7f-6cff-4a38-a3a0-ce490ff37330-kube-api-access-krz54\") pod \"limitador-limitador-84994f7ffc-ql74p\" (UID: \"f5a40e7f-6cff-4a38-a3a0-ce490ff37330\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ql74p"
Sep 15 22:36:09.961026 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.961006    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f5a40e7f-6cff-4a38-a3a0-ce490ff37330-config-file\") pod \"limitador-limitador-84994f7ffc-ql74p\" (UID: \"f5a40e7f-6cff-4a38-a3a0-ce490ff37330\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ql74p"
Sep 15 22:36:09.968798 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:09.968778    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krz54\" (UniqueName: \"kubernetes.io/projected/f5a40e7f-6cff-4a38-a3a0-ce490ff37330-kube-api-access-krz54\") pod \"limitador-limitador-84994f7ffc-ql74p\" (UID: \"f5a40e7f-6cff-4a38-a3a0-ce490ff37330\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ql74p"
Sep 15 22:36:10.128393 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:10.128308    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-ql74p"
Sep 15 22:36:10.267678 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:10.267628    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:10.365211 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:10.365173    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:10.402687 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:10.402640    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:36:10.405033 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:36:10.405008    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a40e7f_6cff_4a38_a3a0_ce490ff37330.slice/crio-eeed8fb514cce9ebdafa04189b0d29e5771d79843b73eb5598de501da23b8ae1 WatchSource:0}: Error finding container eeed8fb514cce9ebdafa04189b0d29e5771d79843b73eb5598de501da23b8ae1: Status 404 returned error can't find the container with id eeed8fb514cce9ebdafa04189b0d29e5771d79843b73eb5598de501da23b8ae1
Sep 15 22:36:10.415171 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:10.415145    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-ql74p" event={"ID":"f5a40e7f-6cff-4a38-a3a0-ce490ff37330","Type":"ContainerStarted","Data":"eeed8fb514cce9ebdafa04189b0d29e5771d79843b73eb5598de501da23b8ae1"}
Sep 15 22:36:10.415525 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:10.415346    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:10.416167 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:10.416150    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:36:11.421597 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:11.421565    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-ql74p" event={"ID":"f5a40e7f-6cff-4a38-a3a0-ce490ff37330","Type":"ContainerStarted","Data":"605fc64a1b5c303a231836bff6e0a9c35348d9ab70608511a8fc288bcf140b3d"}
Sep 15 22:36:11.421892 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:11.421717    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-ql74p"
Sep 15 22:36:11.456841 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:11.456799    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-ql74p" podStartSLOduration=1.58092549 podStartE2EDuration="2.456783641s" totalImagesPullingTime="875.858151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:36:11.456183501 +0000 UTC m=+544.074998408" watchObservedRunningTime="2026-09-15 22:36:11.456783641 +0000 UTC m=+544.075598549"
Sep 15 22:36:17.520935 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:17.520885    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:17.580375 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:17.580337    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:22.428307 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:22.428251    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-ql74p"
Sep 15 22:36:25.581721 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:25.581667    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:25.672286 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:25.672222    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:26.097790 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:26.097750    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:35.478552 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:35.478514    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:35.951423 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:35.951387    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:41.973992 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:41.973953    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:42.186097 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.186062    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54fd547858-8zt6x"]
Sep 15 22:36:42.194497 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.194476    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 22:36:42.201763 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.198823    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 22:36:42.201763 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.199093    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 22:36:42.201763 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.199341    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zbqvm\""
Sep 15 22:36:42.201763 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.199582    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 22:36:42.204887 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.204865    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54fd547858-8zt6x"]
Sep 15 22:36:42.337128 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.337099    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2485cbbf-64ed-406c-b3a3-c64880fe1be9-metrics-certs\") pod \"maas-controller-54fd547858-8zt6x\" (UID: \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\") " pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 22:36:42.337323 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.337137    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2485cbbf-64ed-406c-b3a3-c64880fe1be9-webhook-cert\") pod \"maas-controller-54fd547858-8zt6x\" (UID: \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\") " pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 22:36:42.337323 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.337290    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwx92\" (UniqueName: \"kubernetes.io/projected/2485cbbf-64ed-406c-b3a3-c64880fe1be9-kube-api-access-rwx92\") pod \"maas-controller-54fd547858-8zt6x\" (UID: \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\") " pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 22:36:42.438549 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.438520    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwx92\" (UniqueName: \"kubernetes.io/projected/2485cbbf-64ed-406c-b3a3-c64880fe1be9-kube-api-access-rwx92\") pod \"maas-controller-54fd547858-8zt6x\" (UID: \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\") " pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 22:36:42.438715 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.438566    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2485cbbf-64ed-406c-b3a3-c64880fe1be9-metrics-certs\") pod \"maas-controller-54fd547858-8zt6x\" (UID: \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\") " pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 22:36:42.438715 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.438585    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2485cbbf-64ed-406c-b3a3-c64880fe1be9-webhook-cert\") pod \"maas-controller-54fd547858-8zt6x\" (UID: \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\") " pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 22:36:42.440988 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.440965    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2485cbbf-64ed-406c-b3a3-c64880fe1be9-webhook-cert\") pod \"maas-controller-54fd547858-8zt6x\" (UID: \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\") " pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 22:36:42.441126 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.441102    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2485cbbf-64ed-406c-b3a3-c64880fe1be9-metrics-certs\") pod \"maas-controller-54fd547858-8zt6x\" (UID: \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\") " pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 22:36:42.447678 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.447651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwx92\" (UniqueName: \"kubernetes.io/projected/2485cbbf-64ed-406c-b3a3-c64880fe1be9-kube-api-access-rwx92\") pod \"maas-controller-54fd547858-8zt6x\" (UID: \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\") " pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 22:36:42.513775 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.513748    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 22:36:42.785945 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:42.785920    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54fd547858-8zt6x"]
Sep 15 22:36:42.788002 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:36:42.787972    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2485cbbf_64ed_406c_b3a3_c64880fe1be9.slice/crio-10716b6c2accdee1463cdb43123ca369e208b20f40f4a2420a9b604cc965e1fe WatchSource:0}: Error finding container 10716b6c2accdee1463cdb43123ca369e208b20f40f4a2420a9b604cc965e1fe: Status 404 returned error can't find the container with id 10716b6c2accdee1463cdb43123ca369e208b20f40f4a2420a9b604cc965e1fe
Sep 15 22:36:43.183807 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:43.183771    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:43.564317 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:43.564261    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd547858-8zt6x" event={"ID":"2485cbbf-64ed-406c-b3a3-c64880fe1be9","Type":"ContainerStarted","Data":"10716b6c2accdee1463cdb43123ca369e208b20f40f4a2420a9b604cc965e1fe"}
Sep 15 22:36:45.554628 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:45.554590    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:45.575382 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:45.575346    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd547858-8zt6x" event={"ID":"2485cbbf-64ed-406c-b3a3-c64880fe1be9","Type":"ContainerStarted","Data":"5fd67dd2c285dfdbde827f1d658ee7cb45048e4a618f3ee1699bb6b40d7b1ec6"}
Sep 15 22:36:45.575540 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:45.575474    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 22:36:45.715203 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:45.715122    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:47.958169 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:47.958129    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:48.058782 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:48.058726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:48.957152 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:48.957108    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:36:56.582126 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:56.582098    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 22:36:56.601615 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:36:56.601567    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-54fd547858-8zt6x" podStartSLOduration=12.039326044 podStartE2EDuration="14.601552347s" totalImagesPullingTime="2.562226303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:36:45.629184612 +0000 UTC m=+578.247999517" watchObservedRunningTime="2026-09-15 22:36:56.601552347 +0000 UTC m=+589.220367255"
Sep 15 22:37:07.834169 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:07.834137    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:37:07.836782 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:07.836760    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:37:17.209896 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:17.209850    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:37:17.301922 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:17.301870    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:37:18.707042 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:18.707005    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:37:18.804964 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:18.804919    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:37:18.957993 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:18.957913    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:37:19.259848 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.259758    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:37:19.359353 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.359310    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:37:19.399059 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.399031    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-x6hfn"]
Sep 15 22:37:19.402444 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.402425    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6hfn"
Sep 15 22:37:19.405546 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.405525    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tgjp8\""
Sep 15 22:37:19.407072 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.407050    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 22:37:19.419308 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.419262    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-x6hfn"]
Sep 15 22:37:19.423685 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.423655    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:37:19.530310 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.530227    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwph\" (UniqueName: \"kubernetes.io/projected/af0f1d9d-0d60-4861-85c6-e991c8e58bd7-kube-api-access-2dwph\") pod \"payload-processing-5b54b7fbb7-x6hfn\" (UID: \"af0f1d9d-0d60-4861-85c6-e991c8e58bd7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x6hfn"
Sep 15 22:37:19.530440 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.530352    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af0f1d9d-0d60-4861-85c6-e991c8e58bd7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-x6hfn\" (UID: \"af0f1d9d-0d60-4861-85c6-e991c8e58bd7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x6hfn"
Sep 15 22:37:19.631528 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.631491    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwph\" (UniqueName: \"kubernetes.io/projected/af0f1d9d-0d60-4861-85c6-e991c8e58bd7-kube-api-access-2dwph\") pod \"payload-processing-5b54b7fbb7-x6hfn\" (UID: \"af0f1d9d-0d60-4861-85c6-e991c8e58bd7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x6hfn"
Sep 15 22:37:19.631700 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.631549    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af0f1d9d-0d60-4861-85c6-e991c8e58bd7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-x6hfn\" (UID: \"af0f1d9d-0d60-4861-85c6-e991c8e58bd7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x6hfn"
Sep 15 22:37:19.632245 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.632225    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af0f1d9d-0d60-4861-85c6-e991c8e58bd7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-x6hfn\" (UID: \"af0f1d9d-0d60-4861-85c6-e991c8e58bd7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x6hfn"
Sep 15 22:37:19.641155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.641125    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwph\" (UniqueName: \"kubernetes.io/projected/af0f1d9d-0d60-4861-85c6-e991c8e58bd7-kube-api-access-2dwph\") pod \"payload-processing-5b54b7fbb7-x6hfn\" (UID: \"af0f1d9d-0d60-4861-85c6-e991c8e58bd7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x6hfn"
Sep 15 22:37:19.713383 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.713341    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6hfn"
Sep 15 22:37:19.991553 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:19.991524    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-x6hfn"]
Sep 15 22:37:19.993749 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:37:19.993717    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0f1d9d_0d60_4861_85c6_e991c8e58bd7.slice/crio-3eff1a8ca7dd1af61c22cd0824775023613e2731a7985274278928090b9f3965 WatchSource:0}: Error finding container 3eff1a8ca7dd1af61c22cd0824775023613e2731a7985274278928090b9f3965: Status 404 returned error can't find the container with id 3eff1a8ca7dd1af61c22cd0824775023613e2731a7985274278928090b9f3965
Sep 15 22:37:20.736400 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:20.736314    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6hfn" event={"ID":"af0f1d9d-0d60-4861-85c6-e991c8e58bd7","Type":"ContainerStarted","Data":"3eff1a8ca7dd1af61c22cd0824775023613e2731a7985274278928090b9f3965"}
Sep 15 22:37:23.754314 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:23.754261    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6hfn" event={"ID":"af0f1d9d-0d60-4861-85c6-e991c8e58bd7","Type":"ContainerStarted","Data":"b188d939644f7de649fafa00d377e4e54bedac79d5c633be071fec3fb1d23a05"}
Sep 15 22:37:23.754704 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:23.754391    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6hfn"
Sep 15 22:37:23.773244 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:23.773178    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6hfn" podStartSLOduration=1.927934598 podStartE2EDuration="4.773158316s" totalImagesPullingTime="2.845223718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:37:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:37:23.772181408 +0000 UTC m=+616.390996317" watchObservedRunningTime="2026-09-15 22:37:23.773158316 +0000 UTC m=+616.391973225"
Sep 15 22:37:29.526307 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:29.526249    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:37:29.619986 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:29.619949    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:37:29.783882 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:29.783803    2584 generic.go:356] "Generic (PLEG): container finished" podID="847f9278-fbd9-44ff-b475-62af5997177f" containerID="1544fe57835f43c48af7d26f767803eb3fd08f8416d5588be617b68ab1f829b4" exitCode=0
Sep 15 22:37:29.783882 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:29.783871    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb" event={"ID":"847f9278-fbd9-44ff-b475-62af5997177f","Type":"ContainerDied","Data":"1544fe57835f43c48af7d26f767803eb3fd08f8416d5588be617b68ab1f829b4"}
Sep 15 22:37:29.784237 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:29.784223    2584 scope.go:117] "RemoveContainer" containerID="1544fe57835f43c48af7d26f767803eb3fd08f8416d5588be617b68ab1f829b4"
Sep 15 22:37:29.785346 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:29.785313    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:37:29.785425 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:29.785393    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:37:29.785461 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:29.785426    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:37:30.794046 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:30.793242    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb" event={"ID":"847f9278-fbd9-44ff-b475-62af5997177f","Type":"ContainerStarted","Data":"b3bc3c35f58291f2cda1369d377321115b0bd1a0858ba769f22169b5c9f8bbea"}
Sep 15 22:37:31.547173 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:31.547138    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:37:31.547371 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:31.547259    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8b2xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 15 22:37:31.547371 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:31.547328    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb" podUID="847f9278-fbd9-44ff-b475-62af5997177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 15 22:37:32.547919 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:32.547879    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8b2xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 15 22:37:32.548364 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:32.547937    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb" podUID="847f9278-fbd9-44ff-b475-62af5997177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 15 22:37:33.547034 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:33.546995    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8b2xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 15 22:37:33.547196 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:33.547049    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb" podUID="847f9278-fbd9-44ff-b475-62af5997177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 15 22:37:34.547825 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:34.547789    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8b2xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 15 22:37:34.548221 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:34.547846    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb" podUID="847f9278-fbd9-44ff-b475-62af5997177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 15 22:37:34.761846 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:34.761815    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6hfn"
Sep 15 22:37:35.547762 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:35.547727    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8b2xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 15 22:37:35.547939 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:35.547781    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb" podUID="847f9278-fbd9-44ff-b475-62af5997177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 15 22:37:36.551695 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:36.551655    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:37:36.552084 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:36.551922    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:37:36.552541 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:36.552519    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8b2xb"
Sep 15 22:37:39.967033 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:39.966991    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"]
Sep 15 22:37:39.969405 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:39.969382    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:39.973474 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:39.973451    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 22:37:39.973631 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:39.973613    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 22:37:39.973704 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:39.973524    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 15 22:37:39.973704 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:39.973525    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-582dz\""
Sep 15 22:37:39.982334 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:39.982311    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"]
Sep 15 22:37:40.009111 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.009086    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:37:40.116503 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.116462    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.116503 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.116502    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.116739 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.116541    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6hmb\" (UniqueName: \"kubernetes.io/projected/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-kube-api-access-v6hmb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.116739 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.116592    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.116739 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.116687    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.116739 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.116712    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.217645 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.217562    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.217645 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.217597    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.217645 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.217624    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6hmb\" (UniqueName: \"kubernetes.io/projected/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-kube-api-access-v6hmb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.217898 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.217663    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.217898 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.217705    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.217898 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.217726    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.218062 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.218035    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.218120 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.218072    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.218168 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.218148    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.220021 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.220005    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.220319 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.220298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.226113 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.226089    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6hmb\" (UniqueName: \"kubernetes.io/projected/5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3-kube-api-access-v6hmb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t\" (UID: \"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.280246 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.280221    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:40.400970 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.400935    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:37:40.551028 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.551006    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"]
Sep 15 22:37:40.552538 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:37:40.552507    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecc93c7_3ce3_4f8b_ad13_2966714ac9f3.slice/crio-3d9a6b83fd35e1cf435839371b19cf6d0ce5727190681f52a6b32541c4e1e4a7 WatchSource:0}: Error finding container 3d9a6b83fd35e1cf435839371b19cf6d0ce5727190681f52a6b32541c4e1e4a7: Status 404 returned error can't find the container with id 3d9a6b83fd35e1cf435839371b19cf6d0ce5727190681f52a6b32541c4e1e4a7
Sep 15 22:37:40.844142 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:40.844014    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t" event={"ID":"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3","Type":"ContainerStarted","Data":"3d9a6b83fd35e1cf435839371b19cf6d0ce5727190681f52a6b32541c4e1e4a7"}
Sep 15 22:37:41.850010 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:41.849975    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t" event={"ID":"5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3","Type":"ContainerStarted","Data":"2c9d7d0b2b4012f7e1c2886888535cd88ff50e05bef224e9bade60949da80535"}
Sep 15 22:37:41.869255 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:41.869207    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t" podStartSLOduration=1.924726074 podStartE2EDuration="2.869192336s" totalImagesPullingTime="944.466262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:37:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:37:41.868543345 +0000 UTC m=+634.487358256" watchObservedRunningTime="2026-09-15 22:37:41.869192336 +0000 UTC m=+634.488007238"
Sep 15 22:37:46.903792 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:46.903751    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:37:47.477457 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:47.477423    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:37:50.281151 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:50.281111    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:50.281614 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:50.281186    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:50.294045 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:50.294019    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:50.905978 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:50.905947    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t"
Sep 15 22:37:59.859138 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.859099    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"]
Sep 15 22:37:59.864048 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.864023    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.867356 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.867329    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 15 22:37:59.874067 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.874043    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"]
Sep 15 22:37:59.875697 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.875669    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4babe645-19bd-4280-8420-9084fe9858ce-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.875890 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.875866    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4babe645-19bd-4280-8420-9084fe9858ce-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.876059 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.875927    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4babe645-19bd-4280-8420-9084fe9858ce-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.876059 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.875964    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4babe645-19bd-4280-8420-9084fe9858ce-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.876059 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.875987    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4babe645-19bd-4280-8420-9084fe9858ce-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.876059 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.876017    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58wcx\" (UniqueName: \"kubernetes.io/projected/4babe645-19bd-4280-8420-9084fe9858ce-kube-api-access-58wcx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.899280 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.899242    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:37:59.976530 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.976494    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4babe645-19bd-4280-8420-9084fe9858ce-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.976530 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.976530    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4babe645-19bd-4280-8420-9084fe9858ce-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.976782 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.976546    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4babe645-19bd-4280-8420-9084fe9858ce-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.976782 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.976565    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58wcx\" (UniqueName: \"kubernetes.io/projected/4babe645-19bd-4280-8420-9084fe9858ce-kube-api-access-58wcx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.976782 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.976718    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4babe645-19bd-4280-8420-9084fe9858ce-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.976782 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.976751    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4babe645-19bd-4280-8420-9084fe9858ce-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.977005 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.976979    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4babe645-19bd-4280-8420-9084fe9858ce-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.977063 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.977042    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4babe645-19bd-4280-8420-9084fe9858ce-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.977126 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.977047    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4babe645-19bd-4280-8420-9084fe9858ce-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.979244 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.979221    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4babe645-19bd-4280-8420-9084fe9858ce-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.979783 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.979764    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4babe645-19bd-4280-8420-9084fe9858ce-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:37:59.986010 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:37:59.985982    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58wcx\" (UniqueName: \"kubernetes.io/projected/4babe645-19bd-4280-8420-9084fe9858ce-kube-api-access-58wcx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9\" (UID: \"4babe645-19bd-4280-8420-9084fe9858ce\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:38:00.176473 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:00.176387    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:38:00.446187 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:00.446164    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"]
Sep 15 22:38:00.448390 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:38:00.448364    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4babe645_19bd_4280_8420_9084fe9858ce.slice/crio-7a408bb0a5bd062dc0f9137ff4a9da4131304170bbe1c5c0f3f0a23ea330105f WatchSource:0}: Error finding container 7a408bb0a5bd062dc0f9137ff4a9da4131304170bbe1c5c0f3f0a23ea330105f: Status 404 returned error can't find the container with id 7a408bb0a5bd062dc0f9137ff4a9da4131304170bbe1c5c0f3f0a23ea330105f
Sep 15 22:38:00.764662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:00.764585    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:38:00.937692 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:00.937659    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9" event={"ID":"4babe645-19bd-4280-8420-9084fe9858ce","Type":"ContainerStarted","Data":"b8380abe32753429f4bb51f070362db3d9428c7578e5d4b4d4f935d4fdfaf70d"}
Sep 15 22:38:00.938036 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:00.937705    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9" event={"ID":"4babe645-19bd-4280-8420-9084fe9858ce","Type":"ContainerStarted","Data":"7a408bb0a5bd062dc0f9137ff4a9da4131304170bbe1c5c0f3f0a23ea330105f"}
Sep 15 22:38:00.960199 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:00.960153    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9" podStartSLOduration=1.798269189 podStartE2EDuration="1.960138929s" totalImagesPullingTime="161.86974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:38:00.959291346 +0000 UTC m=+653.578106250" watchObservedRunningTime="2026-09-15 22:38:00.960138929 +0000 UTC m=+653.578953837"
Sep 15 22:38:07.538315 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:07.538248    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:38:10.177462 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:10.177430    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:38:10.177902 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:10.177490    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:38:10.189489 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:10.189466    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:38:10.994203 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:10.994170    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9"
Sep 15 22:38:16.074652 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:16.074613    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:38:17.257513 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:17.257478    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:38:26.302328 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:26.302288    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:38:27.773920 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:27.773886    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:38:41.781812 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:41.781772    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:38:43.660569 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:43.660534    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:38:53.265602 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.265563    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"]
Sep 15 22:38:53.269163 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.269140    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.272094 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.272076    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 22:38:53.278304 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.278249    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"]
Sep 15 22:38:53.292129 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.292103    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:38:53.438117 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.438077    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/280a414e-4228-48e1-881a-7dd9c5358afd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.438117 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.438128    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxlr\" (UniqueName: \"kubernetes.io/projected/280a414e-4228-48e1-881a-7dd9c5358afd-kube-api-access-xpxlr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.438377 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.438193    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/280a414e-4228-48e1-881a-7dd9c5358afd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.438377 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.438225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/280a414e-4228-48e1-881a-7dd9c5358afd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.438377 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.438280    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/280a414e-4228-48e1-881a-7dd9c5358afd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.438377 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.438307    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/280a414e-4228-48e1-881a-7dd9c5358afd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.539427 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.539351    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/280a414e-4228-48e1-881a-7dd9c5358afd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.539427 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.539399    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxlr\" (UniqueName: \"kubernetes.io/projected/280a414e-4228-48e1-881a-7dd9c5358afd-kube-api-access-xpxlr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.539630 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.539442    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/280a414e-4228-48e1-881a-7dd9c5358afd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.539630 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.539475    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/280a414e-4228-48e1-881a-7dd9c5358afd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.539630 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.539515    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/280a414e-4228-48e1-881a-7dd9c5358afd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.539630 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.539539    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/280a414e-4228-48e1-881a-7dd9c5358afd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.539844 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.539802    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/280a414e-4228-48e1-881a-7dd9c5358afd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.539844 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.539821    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/280a414e-4228-48e1-881a-7dd9c5358afd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.540022 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.539999    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/280a414e-4228-48e1-881a-7dd9c5358afd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.541734 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.541715    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/280a414e-4228-48e1-881a-7dd9c5358afd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.541927 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.541907    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/280a414e-4228-48e1-881a-7dd9c5358afd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.547471 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.547436    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxlr\" (UniqueName: \"kubernetes.io/projected/280a414e-4228-48e1-881a-7dd9c5358afd-kube-api-access-xpxlr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz\" (UID: \"280a414e-4228-48e1-881a-7dd9c5358afd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.579423 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.579402    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:38:53.853784 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.853758    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"]
Sep 15 22:38:53.855627 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:38:53.855598    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod280a414e_4228_48e1_881a_7dd9c5358afd.slice/crio-bbcc302a6fdd63c5c88b424bed2bfc4d42a5fcab31e7a396f0f3f1a402b688e4 WatchSource:0}: Error finding container bbcc302a6fdd63c5c88b424bed2bfc4d42a5fcab31e7a396f0f3f1a402b688e4: Status 404 returned error can't find the container with id bbcc302a6fdd63c5c88b424bed2bfc4d42a5fcab31e7a396f0f3f1a402b688e4
Sep 15 22:38:53.857061 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:53.857044    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 22:38:54.180774 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:54.180686    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz" event={"ID":"280a414e-4228-48e1-881a-7dd9c5358afd","Type":"ContainerStarted","Data":"b46b3009f0075404a95ebf60006ca061daff87e63499f4e6f7c85578ba58f19f"}
Sep 15 22:38:54.180774 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:54.180745    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz" event={"ID":"280a414e-4228-48e1-881a-7dd9c5358afd","Type":"ContainerStarted","Data":"bbcc302a6fdd63c5c88b424bed2bfc4d42a5fcab31e7a396f0f3f1a402b688e4"}
Sep 15 22:38:54.198252 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:54.198205    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz" podStartSLOduration=1.018889361 podStartE2EDuration="1.198191727s" totalImagesPullingTime="179.302366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:38:54.197243049 +0000 UTC m=+706.816057958" watchObservedRunningTime="2026-09-15 22:38:54.198191727 +0000 UTC m=+706.817006634"
Sep 15 22:38:55.264852 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:38:55.264821    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:39:03.580416 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:39:03.580325    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:39:03.580416 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:39:03.580393    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:39:03.592740 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:39:03.592715    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:39:04.237953 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:39:04.237922    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz"
Sep 15 22:39:56.342529 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:39:56.342483    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:05.903286 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:05.903238    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:06.002676 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:06.002636    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:07.012760 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:07.012709    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:07.106147 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:07.106115    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:22.474439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:22.474401    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:22.535948 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:22.535912    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:22.607192 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:22.607155    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:22.842593 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:22.842557    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:22.880563 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:22.880527    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"]
Sep 15 22:40:22.884635 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:22.884610    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:22.887767 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:22.887732    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fffdae-openshift-default-dockercfg-s9gpz\""
Sep 15 22:40:22.896650 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:22.896623    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"]
Sep 15 22:40:22.960670 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:22.960638    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:22.997466 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:22.997431    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"]
Sep 15 22:40:23.000938 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.000907    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-data\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.001088 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.000964    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2zvk\" (UniqueName: \"kubernetes.io/projected/5bda438d-af76-4c2d-8b4e-f39d13c857ea-kube-api-access-f2zvk\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.001088 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.001050    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-token\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.001184 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.001096    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-workload-socket\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.001184 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.001165    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-credential-socket\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.001295 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.001206    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-workload-certs\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.001358 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.001327    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-podinfo\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.001410 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.001368    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istiod-ca-cert\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.001460 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.001424    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.001509 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.001427    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-envoy\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.004867 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.004842    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fe020d-openshift-default-dockercfg-ssvf9\""
Sep 15 22:40:23.030464 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.030431    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"]
Sep 15 22:40:23.102987 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.102909    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-data\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.102987 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.102946    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istiod-ca-cert\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.102987 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.102969    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-data\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.103255 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103002    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2zvk\" (UniqueName: \"kubernetes.io/projected/5bda438d-af76-4c2d-8b4e-f39d13c857ea-kube-api-access-f2zvk\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.103255 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103046    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-token\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.103255 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-workload-socket\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.103255 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103193    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-workload-certs\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.103255 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103239    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-envoy\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.103526 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103331    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-credential-socket\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.103526 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103383    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-data\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.103526 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103390    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-workload-socket\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.103526 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103434    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-workload-socket\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.103526 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-workload-certs\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.103526 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103515    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-credential-socket\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.103789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103540    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-token\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.103789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103570    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-podinfo\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.103789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103602    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-credential-socket\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.103789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103621    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istiod-ca-cert\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.103789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103678    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-envoy\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.103789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103688    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-workload-certs\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.103789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103705    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-podinfo\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.103789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.103745    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb9hx\" (UniqueName: \"kubernetes.io/projected/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-kube-api-access-nb9hx\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.104202 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.104178    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istiod-ca-cert\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.105878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.105859    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-envoy\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.106131 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.106114    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-podinfo\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.120408 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.120385    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2zvk\" (UniqueName: \"kubernetes.io/projected/5bda438d-af76-4c2d-8b4e-f39d13c857ea-kube-api-access-f2zvk\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.120608 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.120590    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-token\") pod \"e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") " pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.128608 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.128503    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:23.201279 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.201247    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:23.204574 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.204548    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-workload-socket\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.204732 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.204593    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-credential-socket\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.204732 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.204612    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-token\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.204732 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.204669    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-podinfo\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.204732 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.204698    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9hx\" (UniqueName: \"kubernetes.io/projected/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-kube-api-access-nb9hx\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.204949 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.204741    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istiod-ca-cert\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.204949 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.204766    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-data\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.204949 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.204817    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-workload-certs\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.204949 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.204843    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-envoy\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.205157 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.205007    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-workload-socket\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.205293 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.205114    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-credential-socket\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.205926 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.205545    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-data\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.205926 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.205575    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istiod-ca-cert\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.206521 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.206247    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-workload-certs\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.207588 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.207559    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-envoy\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.207788 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.207768    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-podinfo\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.213837 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.213812    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-token\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.214330 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.214307    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb9hx\" (UniqueName: \"kubernetes.io/projected/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-kube-api-access-nb9hx\") pod \"e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") " pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.220966 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.220924    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:23.313334 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.313296    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:23.321617 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.321588    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:23.488154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.488127    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"]
Sep 15 22:40:23.489988 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:40:23.489961    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bda438d_af76_4c2d_8b4e_f39d13c857ea.slice/crio-4eb38a0fafe098fbd274855447f0985d5ec1401a693a5c376b2b5a19da56bd6e WatchSource:0}: Error finding container 4eb38a0fafe098fbd274855447f0985d5ec1401a693a5c376b2b5a19da56bd6e: Status 404 returned error can't find the container with id 4eb38a0fafe098fbd274855447f0985d5ec1401a693a5c376b2b5a19da56bd6e
Sep 15 22:40:23.492162 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.492127    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:40:23.492252 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.492189    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:40:23.492252 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.492219    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:40:23.591899 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.591872    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"]
Sep 15 22:40:23.596749 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.596706    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:40:23.596855 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.596795    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:40:23.596855 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.596845    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:40:23.597478 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.597447    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw" event={"ID":"5bda438d-af76-4c2d-8b4e-f39d13c857ea","Type":"ContainerStarted","Data":"570bbb34c381c0c606cf5a9d6e022dca0e024d4aaaaa9faac05da476d607274b"}
Sep 15 22:40:23.597575 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.597501    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw" event={"ID":"5bda438d-af76-4c2d-8b4e-f39d13c857ea","Type":"ContainerStarted","Data":"4eb38a0fafe098fbd274855447f0985d5ec1401a693a5c376b2b5a19da56bd6e"}
Sep 15 22:40:23.621075 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.621030    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw" podStartSLOduration=1.621013193 podStartE2EDuration="1.621013193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:40:23.619772304 +0000 UTC m=+796.238587213" watchObservedRunningTime="2026-09-15 22:40:23.621013193 +0000 UTC m=+796.239828160"
Sep 15 22:40:23.906334 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:23.906206    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:24.008642 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:24.008608    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:24.201866 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:24.201782    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:24.207047 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:24.207023    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:24.359290 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:24.359232    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:24.608452 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:24.608406    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" event={"ID":"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca","Type":"ContainerStarted","Data":"c48c3ba34829c989333d521e47efca8c4f0f28a144ff85afcd020b29a8d25e95"}
Sep 15 22:40:24.608912 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:24.608847    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:24.608988 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:24.608916    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" event={"ID":"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca","Type":"ContainerStarted","Data":"3147ee17c7fb05a633d55b1f998b8f4537c171588bc2bc91ce2b7863473e686b"}
Sep 15 22:40:24.611274 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:24.611244    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:24.629913 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:24.629867    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" podStartSLOduration=2.629852764 podStartE2EDuration="2.629852764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:40:24.628714275 +0000 UTC m=+797.247529185" watchObservedRunningTime="2026-09-15 22:40:24.629852764 +0000 UTC m=+797.248667671"
Sep 15 22:40:25.314210 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:25.314174    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:25.318924 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:25.318897    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:25.592642 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:25.592549    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:25.611966 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:25.611930    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:25.612948 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:25.612930    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:25.694647 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:25.694604    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:25.792033 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:25.791993    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:26.011809 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:26.011707    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:26.106525 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:26.106483    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:26.363596 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:26.363560    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:40:26.626915 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:26.626825    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:26.718545 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:26.718512    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:26.974532 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:26.974442    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:27.074996 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.074962    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:27.278142 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.278057    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:27.568940 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.568906    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:27.668859 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.668817    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:27.711868 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.711834    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"]
Sep 15 22:40:27.715651 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.715627    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"
Sep 15 22:40:27.718394 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.718367    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fffdae\""
Sep 15 22:40:27.718394 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.718392    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fffdae-dockercfg-m6tcd\""
Sep 15 22:40:27.726906 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.726770    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"]
Sep 15 22:40:27.845886 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.845793    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922688b5-15dc-40c6-a4b1-be53e2ba7881-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j\" (UID: \"922688b5-15dc-40c6-a4b1-be53e2ba7881\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"
Sep 15 22:40:27.846045 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.845899    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4sd\" (UniqueName: \"kubernetes.io/projected/922688b5-15dc-40c6-a4b1-be53e2ba7881-kube-api-access-mv4sd\") pod \"payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j\" (UID: \"922688b5-15dc-40c6-a4b1-be53e2ba7881\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"
Sep 15 22:40:27.946828 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.946793    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4sd\" (UniqueName: \"kubernetes.io/projected/922688b5-15dc-40c6-a4b1-be53e2ba7881-kube-api-access-mv4sd\") pod \"payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j\" (UID: \"922688b5-15dc-40c6-a4b1-be53e2ba7881\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"
Sep 15 22:40:27.947006 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.946848    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922688b5-15dc-40c6-a4b1-be53e2ba7881-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j\" (UID: \"922688b5-15dc-40c6-a4b1-be53e2ba7881\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"
Sep 15 22:40:27.947798 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.947779    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922688b5-15dc-40c6-a4b1-be53e2ba7881-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j\" (UID: \"922688b5-15dc-40c6-a4b1-be53e2ba7881\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"
Sep 15 22:40:27.957219 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:27.957194    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4sd\" (UniqueName: \"kubernetes.io/projected/922688b5-15dc-40c6-a4b1-be53e2ba7881-kube-api-access-mv4sd\") pod \"payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j\" (UID: \"922688b5-15dc-40c6-a4b1-be53e2ba7881\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"
Sep 15 22:40:28.029575 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.029534    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"
Sep 15 22:40:28.205958 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.205865    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:28.271404 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.271352    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:28.317326 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.317222    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"]
Sep 15 22:40:28.320689 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:40:28.320663    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922688b5_15dc_40c6_a4b1_be53e2ba7881.slice/crio-c9a4a79b6c8ccd77d1cbc5dffac089f37599a379db0f5f23c8df45805a1f5184 WatchSource:0}: Error finding container c9a4a79b6c8ccd77d1cbc5dffac089f37599a379db0f5f23c8df45805a1f5184: Status 404 returned error can't find the container with id c9a4a79b6c8ccd77d1cbc5dffac089f37599a379db0f5f23c8df45805a1f5184
Sep 15 22:40:28.341030 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.340998    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:28.631641 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.631605    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j" event={"ID":"922688b5-15dc-40c6-a4b1-be53e2ba7881","Type":"ContainerStarted","Data":"69fa88e36c65ba85a4840311402b2ca736b33cdc63c7ea090560a08fc31727e2"}
Sep 15 22:40:28.631816 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.631664    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j" event={"ID":"922688b5-15dc-40c6-a4b1-be53e2ba7881","Type":"ContainerStarted","Data":"c9a4a79b6c8ccd77d1cbc5dffac089f37599a379db0f5f23c8df45805a1f5184"}
Sep 15 22:40:28.631816 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.631694    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"
Sep 15 22:40:28.652123 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.652067    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j" podStartSLOduration=1.6520504489999999 podStartE2EDuration="1.652050449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:40:28.650714669 +0000 UTC m=+801.269529577" watchObservedRunningTime="2026-09-15 22:40:28.652050449 +0000 UTC m=+801.270865356"
Sep 15 22:40:28.827828 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.827782    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:28.929455 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.929360    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:28.941735 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.941703    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"]
Sep 15 22:40:28.945615 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.945587    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"
Sep 15 22:40:28.948481 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.948461    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-03f258\""
Sep 15 22:40:28.955416 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.955389    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"]
Sep 15 22:40:28.962762 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.962737    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"]
Sep 15 22:40:28.966220 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.966201    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"
Sep 15 22:40:28.969135 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.969115    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-03f258-dockercfg-9dmvc\""
Sep 15 22:40:28.978996 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:28.978967    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"]
Sep 15 22:40:29.058309 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.058278    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2d55aa-1749-434c-947f-9cdbc6a76375-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd\" (UID: \"5a2d55aa-1749-434c-947f-9cdbc6a76375\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"
Sep 15 22:40:29.058480 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.058314    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h45sm\" (UniqueName: \"kubernetes.io/projected/5a2d55aa-1749-434c-947f-9cdbc6a76375-kube-api-access-h45sm\") pod \"payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd\" (UID: \"5a2d55aa-1749-434c-947f-9cdbc6a76375\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"
Sep 15 22:40:29.159784 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.159748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnhgr\" (UniqueName: \"kubernetes.io/projected/aa43a50f-3be7-464c-836b-5554361e62ea-kube-api-access-gnhgr\") pod \"payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd\" (UID: \"aa43a50f-3be7-464c-836b-5554361e62ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"
Sep 15 22:40:29.159995 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.159796    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2d55aa-1749-434c-947f-9cdbc6a76375-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd\" (UID: \"5a2d55aa-1749-434c-947f-9cdbc6a76375\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"
Sep 15 22:40:29.159995 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.159829    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h45sm\" (UniqueName: \"kubernetes.io/projected/5a2d55aa-1749-434c-947f-9cdbc6a76375-kube-api-access-h45sm\") pod \"payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd\" (UID: \"5a2d55aa-1749-434c-947f-9cdbc6a76375\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"
Sep 15 22:40:29.159995 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.159948    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa43a50f-3be7-464c-836b-5554361e62ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd\" (UID: \"aa43a50f-3be7-464c-836b-5554361e62ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"
Sep 15 22:40:29.160480 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.160461    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2d55aa-1749-434c-947f-9cdbc6a76375-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd\" (UID: \"5a2d55aa-1749-434c-947f-9cdbc6a76375\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"
Sep 15 22:40:29.164196 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.164168    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:29.168962 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.168941    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h45sm\" (UniqueName: \"kubernetes.io/projected/5a2d55aa-1749-434c-947f-9cdbc6a76375-kube-api-access-h45sm\") pod \"payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd\" (UID: \"5a2d55aa-1749-434c-947f-9cdbc6a76375\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"
Sep 15 22:40:29.257971 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.257888    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"
Sep 15 22:40:29.261385 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.260908    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa43a50f-3be7-464c-836b-5554361e62ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd\" (UID: \"aa43a50f-3be7-464c-836b-5554361e62ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"
Sep 15 22:40:29.261385 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.260952    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnhgr\" (UniqueName: \"kubernetes.io/projected/aa43a50f-3be7-464c-836b-5554361e62ea-kube-api-access-gnhgr\") pod \"payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd\" (UID: \"aa43a50f-3be7-464c-836b-5554361e62ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"
Sep 15 22:40:29.261679 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.261655    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa43a50f-3be7-464c-836b-5554361e62ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd\" (UID: \"aa43a50f-3be7-464c-836b-5554361e62ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"
Sep 15 22:40:29.270003 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.269977    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnhgr\" (UniqueName: \"kubernetes.io/projected/aa43a50f-3be7-464c-836b-5554361e62ea-kube-api-access-gnhgr\") pod \"payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd\" (UID: \"aa43a50f-3be7-464c-836b-5554361e62ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"
Sep 15 22:40:29.278636 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.278611    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"
Sep 15 22:40:29.437374 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.437297    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:29.508697 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.508605    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:29.540727 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.540520    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"]
Sep 15 22:40:29.543127 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:40:29.543097    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2d55aa_1749_434c_947f_9cdbc6a76375.slice/crio-6039eb60a6ffe489c05004ab517695b65398db4e62ad619393e9d68ac3c27d4b WatchSource:0}: Error finding container 6039eb60a6ffe489c05004ab517695b65398db4e62ad619393e9d68ac3c27d4b: Status 404 returned error can't find the container with id 6039eb60a6ffe489c05004ab517695b65398db4e62ad619393e9d68ac3c27d4b
Sep 15 22:40:29.577095 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.577075    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"]
Sep 15 22:40:29.579895 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:40:29.579866    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa43a50f_3be7_464c_836b_5554361e62ea.slice/crio-d756cc57f137d1b49ca80f57ca5c8d6d2558a49eff1f86eefc1cd25a9cfafc99 WatchSource:0}: Error finding container d756cc57f137d1b49ca80f57ca5c8d6d2558a49eff1f86eefc1cd25a9cfafc99: Status 404 returned error can't find the container with id d756cc57f137d1b49ca80f57ca5c8d6d2558a49eff1f86eefc1cd25a9cfafc99
Sep 15 22:40:29.581753 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.581725    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:29.640736 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.640443    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd" event={"ID":"5a2d55aa-1749-434c-947f-9cdbc6a76375","Type":"ContainerStarted","Data":"e8ce1ccd58ee1dbdd35d2012fae78517bed816b7a9a01c8c2077741a227fbc27"}
Sep 15 22:40:29.640736 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.640507    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd" event={"ID":"5a2d55aa-1749-434c-947f-9cdbc6a76375","Type":"ContainerStarted","Data":"6039eb60a6ffe489c05004ab517695b65398db4e62ad619393e9d68ac3c27d4b"}
Sep 15 22:40:29.640736 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.640645    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"
Sep 15 22:40:29.642765 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.642651    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd" event={"ID":"aa43a50f-3be7-464c-836b-5554361e62ea","Type":"ContainerStarted","Data":"e1f3ee3da179a5dc2b63242c6b7de84407b2fb7246064d556060d2961c8fd3f5"}
Sep 15 22:40:29.642765 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.642709    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd" event={"ID":"aa43a50f-3be7-464c-836b-5554361e62ea","Type":"ContainerStarted","Data":"d756cc57f137d1b49ca80f57ca5c8d6d2558a49eff1f86eefc1cd25a9cfafc99"}
Sep 15 22:40:29.643139 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.643117    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"
Sep 15 22:40:29.661976 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.661935    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd" podStartSLOduration=1.6619199070000001 podStartE2EDuration="1.661919907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:40:29.659699575 +0000 UTC m=+802.278514484" watchObservedRunningTime="2026-09-15 22:40:29.661919907 +0000 UTC m=+802.280734815"
Sep 15 22:40:29.677327 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.677056    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd" podStartSLOduration=1.677038198 podStartE2EDuration="1.677038198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:40:29.676395873 +0000 UTC m=+802.295210782" watchObservedRunningTime="2026-09-15 22:40:29.677038198 +0000 UTC m=+802.295853110"
Sep 15 22:40:29.901928 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:29.901882    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:30.040903 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.040867    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:30.142539 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.142504    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:30.143797 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.143775    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"]
Sep 15 22:40:30.147329 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.147310    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"
Sep 15 22:40:30.149900 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.149856    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-fe020d\""
Sep 15 22:40:30.150024 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.149977    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-fe020d-dockercfg-5d8d6\""
Sep 15 22:40:30.157947 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.157924    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"]
Sep 15 22:40:30.271780 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.271676    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvcvz\" (UniqueName: \"kubernetes.io/projected/22cf15a7-2d61-4507-a202-27a77939e4d2-kube-api-access-nvcvz\") pod \"payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j\" (UID: \"22cf15a7-2d61-4507-a202-27a77939e4d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"
Sep 15 22:40:30.271976 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.271942    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cf15a7-2d61-4507-a202-27a77939e4d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j\" (UID: \"22cf15a7-2d61-4507-a202-27a77939e4d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"
Sep 15 22:40:30.373558 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.373519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvcvz\" (UniqueName: \"kubernetes.io/projected/22cf15a7-2d61-4507-a202-27a77939e4d2-kube-api-access-nvcvz\") pod \"payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j\" (UID: \"22cf15a7-2d61-4507-a202-27a77939e4d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"
Sep 15 22:40:30.373756 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.373673    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cf15a7-2d61-4507-a202-27a77939e4d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j\" (UID: \"22cf15a7-2d61-4507-a202-27a77939e4d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"
Sep 15 22:40:30.374489 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.374468    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cf15a7-2d61-4507-a202-27a77939e4d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j\" (UID: \"22cf15a7-2d61-4507-a202-27a77939e4d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"
Sep 15 22:40:30.381540 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.381510    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvcvz\" (UniqueName: \"kubernetes.io/projected/22cf15a7-2d61-4507-a202-27a77939e4d2-kube-api-access-nvcvz\") pod \"payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j\" (UID: \"22cf15a7-2d61-4507-a202-27a77939e4d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"
Sep 15 22:40:30.461321 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.461232    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"
Sep 15 22:40:30.485905 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.485872    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:30.743671 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:30.743645    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"]
Sep 15 22:40:30.745968 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:40:30.745930    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cf15a7_2d61_4507_a202_27a77939e4d2.slice/crio-53f6dc7081b9ecee561284ca4fc985d4452598e8bf195aa111195bb13defba38 WatchSource:0}: Error finding container 53f6dc7081b9ecee561284ca4fc985d4452598e8bf195aa111195bb13defba38: Status 404 returned error can't find the container with id 53f6dc7081b9ecee561284ca4fc985d4452598e8bf195aa111195bb13defba38
Sep 15 22:40:31.307958 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:31.307916    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:31.387486 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:31.387442    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:31.487653 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:31.487603    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:31.560092 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:31.559996    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:31.655204 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:31.655167    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j" event={"ID":"22cf15a7-2d61-4507-a202-27a77939e4d2","Type":"ContainerStarted","Data":"1cc84c6a5ea369ef6be5e30e5a5e0b8c3ee178a00fa4963f129137e3cf3f3e35"}
Sep 15 22:40:31.655383 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:31.655248    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j" event={"ID":"22cf15a7-2d61-4507-a202-27a77939e4d2","Type":"ContainerStarted","Data":"53f6dc7081b9ecee561284ca4fc985d4452598e8bf195aa111195bb13defba38"}
Sep 15 22:40:31.655383 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:31.655299    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"
Sep 15 22:40:31.675950 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:31.675892    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j" podStartSLOduration=1.6758759300000001 podStartE2EDuration="1.67587593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:40:31.674143039 +0000 UTC m=+804.292957942" watchObservedRunningTime="2026-09-15 22:40:31.67587593 +0000 UTC m=+804.294690838"
Sep 15 22:40:31.839797 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:31.838924    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:32.157894 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:32.157813    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:32.469540 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:32.469443    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:33.073840 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:33.073801    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:33.164640 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:33.164605    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:33.390727 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:33.390629    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:34.596792 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:34.596756    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:34.691707 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:34.691660    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:36.103132 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:36.103088    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:37.365868 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:37.365822    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:37.469673 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:37.469636    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:37.566577 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:37.566535    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:38.094292 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.093868    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:38.200397 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.200362    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:38.329536 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.329498    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"]
Sep 15 22:40:38.332549 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.332527    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"
Sep 15 22:40:38.335489 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.335466    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-33ff1a95-dockercfg-cvzwt\""
Sep 15 22:40:38.335595 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.335470    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-33ff1a95\""
Sep 15 22:40:38.341004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.340984    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"]
Sep 15 22:40:38.350796 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.350736    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zdmq\" (UniqueName: \"kubernetes.io/projected/d387ab83-e4df-413a-9341-9b314b9a3947-kube-api-access-5zdmq\") pod \"payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf\" (UID: \"d387ab83-e4df-413a-9341-9b314b9a3947\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"
Sep 15 22:40:38.350796 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.350784    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d387ab83-e4df-413a-9341-9b314b9a3947-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf\" (UID: \"d387ab83-e4df-413a-9341-9b314b9a3947\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"
Sep 15 22:40:38.396141 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.396099    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"]
Sep 15 22:40:38.400498 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.400474    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"
Sep 15 22:40:38.414762 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.414609    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"]
Sep 15 22:40:38.451878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.451846    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zdmq\" (UniqueName: \"kubernetes.io/projected/d387ab83-e4df-413a-9341-9b314b9a3947-kube-api-access-5zdmq\") pod \"payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf\" (UID: \"d387ab83-e4df-413a-9341-9b314b9a3947\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"
Sep 15 22:40:38.452080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.452065    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d387ab83-e4df-413a-9341-9b314b9a3947-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf\" (UID: \"d387ab83-e4df-413a-9341-9b314b9a3947\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"
Sep 15 22:40:38.452194 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.452182    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7bjg\" (UniqueName: \"kubernetes.io/projected/26ba2ee6-ae70-46d7-aded-9d54ed80e11d-kube-api-access-d7bjg\") pod \"payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v\" (UID: \"26ba2ee6-ae70-46d7-aded-9d54ed80e11d\") " pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"
Sep 15 22:40:38.452290 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.452279    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ba2ee6-ae70-46d7-aded-9d54ed80e11d-plugins-config-volume\") pod \"payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v\" (UID: \"26ba2ee6-ae70-46d7-aded-9d54ed80e11d\") " pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"
Sep 15 22:40:38.452877 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.452849    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d387ab83-e4df-413a-9341-9b314b9a3947-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf\" (UID: \"d387ab83-e4df-413a-9341-9b314b9a3947\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"
Sep 15 22:40:38.460532 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.460500    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zdmq\" (UniqueName: \"kubernetes.io/projected/d387ab83-e4df-413a-9341-9b314b9a3947-kube-api-access-5zdmq\") pod \"payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf\" (UID: \"d387ab83-e4df-413a-9341-9b314b9a3947\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"
Sep 15 22:40:38.553093 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.553056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ba2ee6-ae70-46d7-aded-9d54ed80e11d-plugins-config-volume\") pod \"payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v\" (UID: \"26ba2ee6-ae70-46d7-aded-9d54ed80e11d\") " pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"
Sep 15 22:40:38.553322 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.553205    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7bjg\" (UniqueName: \"kubernetes.io/projected/26ba2ee6-ae70-46d7-aded-9d54ed80e11d-kube-api-access-d7bjg\") pod \"payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v\" (UID: \"26ba2ee6-ae70-46d7-aded-9d54ed80e11d\") " pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"
Sep 15 22:40:38.554099 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.554076    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ba2ee6-ae70-46d7-aded-9d54ed80e11d-plugins-config-volume\") pod \"payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v\" (UID: \"26ba2ee6-ae70-46d7-aded-9d54ed80e11d\") " pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"
Sep 15 22:40:38.563014 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.562992    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7bjg\" (UniqueName: \"kubernetes.io/projected/26ba2ee6-ae70-46d7-aded-9d54ed80e11d-kube-api-access-d7bjg\") pod \"payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v\" (UID: \"26ba2ee6-ae70-46d7-aded-9d54ed80e11d\") " pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"
Sep 15 22:40:38.644960 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.644868    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"
Sep 15 22:40:38.665314 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.665284    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:38.729656 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.729621    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"
Sep 15 22:40:38.966183 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:38.966153    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"]
Sep 15 22:40:38.967121 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:40:38.967098    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd387ab83_e4df_413a_9341_9b314b9a3947.slice/crio-e533f5c8e6c3a7b5b662f34b91ce2328011d950dee8d54ec9a129a8534d99230 WatchSource:0}: Error finding container e533f5c8e6c3a7b5b662f34b91ce2328011d950dee8d54ec9a129a8534d99230: Status 404 returned error can't find the container with id e533f5c8e6c3a7b5b662f34b91ce2328011d950dee8d54ec9a129a8534d99230
Sep 15 22:40:39.067197 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:39.067170    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"]
Sep 15 22:40:39.070601 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:40:39.070567    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ba2ee6_ae70_46d7_aded_9d54ed80e11d.slice/crio-423ed18f784359c311417f79e1358a003032136b475a4be15bf9e8eb6fbece45 WatchSource:0}: Error finding container 423ed18f784359c311417f79e1358a003032136b475a4be15bf9e8eb6fbece45: Status 404 returned error can't find the container with id 423ed18f784359c311417f79e1358a003032136b475a4be15bf9e8eb6fbece45
Sep 15 22:40:39.645933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:39.645892    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"
Sep 15 22:40:39.720956 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:39.720919    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v" event={"ID":"26ba2ee6-ae70-46d7-aded-9d54ed80e11d","Type":"ContainerStarted","Data":"177c40a681eebfafed42886493c3d05686b5f0e9890275dd32b552be224e82c6"}
Sep 15 22:40:39.721117 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:39.720985    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v" event={"ID":"26ba2ee6-ae70-46d7-aded-9d54ed80e11d","Type":"ContainerStarted","Data":"423ed18f784359c311417f79e1358a003032136b475a4be15bf9e8eb6fbece45"}
Sep 15 22:40:39.721832 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:39.721807    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"
Sep 15 22:40:39.733325 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:39.732083    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf" event={"ID":"d387ab83-e4df-413a-9341-9b314b9a3947","Type":"ContainerStarted","Data":"4a048f49a4cad04bd461ebefea68e6c1e2b097117ce465651b0aa08fa013e215"}
Sep 15 22:40:39.733325 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:39.732137    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf" event={"ID":"d387ab83-e4df-413a-9341-9b314b9a3947","Type":"ContainerStarted","Data":"e533f5c8e6c3a7b5b662f34b91ce2328011d950dee8d54ec9a129a8534d99230"}
Sep 15 22:40:39.733325 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:39.732709    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"
Sep 15 22:40:39.769809 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:39.769756    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v" podStartSLOduration=1.7697399520000001 podStartE2EDuration="1.769739952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:40:39.750645078 +0000 UTC m=+812.369459979" watchObservedRunningTime="2026-09-15 22:40:39.769739952 +0000 UTC m=+812.388554912"
Sep 15 22:40:39.771210 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:39.771162    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf" podStartSLOduration=1.771153053 podStartE2EDuration="1.771153053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:40:39.769187338 +0000 UTC m=+812.388002259" watchObservedRunningTime="2026-09-15 22:40:39.771153053 +0000 UTC m=+812.389968008"
Sep 15 22:40:39.937118 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:39.937025    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:40.028780 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.028739    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:40.513984 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.513950    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:40.637881 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.637839    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:40.650360 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.650262    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"
Sep 15 22:40:40.650723 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.650471    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"
Sep 15 22:40:40.740112 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.739732    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:40.798920 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.791403    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"]
Sep 15 22:40:40.800434 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.800406    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"
Sep 15 22:40:40.805057 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.804894    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-b08870\""
Sep 15 22:40:40.805661 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.805638    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-b08870-dockercfg-nq2pt\""
Sep 15 22:40:40.807525 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.806880    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"]
Sep 15 22:40:40.839050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.839017    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:40.878118 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.878085    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2-plugins-config-volume\") pod \"payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj\" (UID: \"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2\") " pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"
Sep 15 22:40:40.878315 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.878148    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czc9m\" (UniqueName: \"kubernetes.io/projected/8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2-kube-api-access-czc9m\") pod \"payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj\" (UID: \"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2\") " pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"
Sep 15 22:40:40.979358 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.979314    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2-plugins-config-volume\") pod \"payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj\" (UID: \"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2\") " pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"
Sep 15 22:40:40.979542 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.979372    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czc9m\" (UniqueName: \"kubernetes.io/projected/8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2-kube-api-access-czc9m\") pod \"payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj\" (UID: \"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2\") " pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"
Sep 15 22:40:40.980315 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.980259    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2-plugins-config-volume\") pod \"payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj\" (UID: \"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2\") " pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"
Sep 15 22:40:40.999969 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:40.988259    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czc9m\" (UniqueName: \"kubernetes.io/projected/8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2-kube-api-access-czc9m\") pod \"payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj\" (UID: \"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2\") " pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"
Sep 15 22:40:41.086738 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.086649    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:41.121634 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.121600    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"
Sep 15 22:40:41.181505 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.180924    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:41.281375 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.280796    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:41.401922 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.401486    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"]
Sep 15 22:40:41.403292 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:40:41.403237    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8832304b_b9c5_4b76_9d84_5f4ab3e1ccc2.slice/crio-fff4da9379c8f95332116e8725744934eef8f44601782b8881092ec5104a7840 WatchSource:0}: Error finding container fff4da9379c8f95332116e8725744934eef8f44601782b8881092ec5104a7840: Status 404 returned error can't find the container with id fff4da9379c8f95332116e8725744934eef8f44601782b8881092ec5104a7840
Sep 15 22:40:41.653529 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.653437    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 22:40:41.653977 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.653537    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 22:40:41.661986 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.661941    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:41.749328 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.749279    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj" event={"ID":"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2","Type":"ContainerStarted","Data":"ce50807c030c7f37dc5fb1f4fd87ea488ed00414b8b278504e7f3284997646b3"}
Sep 15 22:40:41.749530 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.749341    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj" event={"ID":"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2","Type":"ContainerStarted","Data":"fff4da9379c8f95332116e8725744934eef8f44601782b8881092ec5104a7840"}
Sep 15 22:40:41.749873 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.749835    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"
Sep 15 22:40:41.781187 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.781124    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj" podStartSLOduration=1.781102378 podStartE2EDuration="1.781102378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:40:41.780207111 +0000 UTC m=+814.399022020" watchObservedRunningTime="2026-09-15 22:40:41.781102378 +0000 UTC m=+814.399917287"
Sep 15 22:40:41.787516 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.787473    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:41.890751 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.890675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:41.915343 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:41.915223    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:40:42.473449 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:42.473406    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:42.662345 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:42.662314    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"
Sep 15 22:40:43.434304 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:43.434235    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:43.616510 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:43.616476    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:43.763981 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:43.763904    2584 generic.go:356] "Generic (PLEG): container finished" podID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerID="570bbb34c381c0c606cf5a9d6e022dca0e024d4aaaaa9faac05da476d607274b" exitCode=0
Sep 15 22:40:43.764378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:43.763979    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw" event={"ID":"5bda438d-af76-4c2d-8b4e-f39d13c857ea","Type":"ContainerDied","Data":"570bbb34c381c0c606cf5a9d6e022dca0e024d4aaaaa9faac05da476d607274b"}
Sep 15 22:40:43.764439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:43.764429    2584 scope.go:117] "RemoveContainer" containerID="570bbb34c381c0c606cf5a9d6e022dca0e024d4aaaaa9faac05da476d607274b"
Sep 15 22:40:43.765632 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:43.765606    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:40:43.765694 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:43.765676    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:40:43.765735 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:43.765707    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:40:43.811090 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:43.811058    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:43.903876 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:43.903835    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:44.004353 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:44.004315    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:44.488290 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:44.488172    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:44.771452 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:44.771372    2584 generic.go:356] "Generic (PLEG): container finished" podID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerID="c48c3ba34829c989333d521e47efca8c4f0f28a144ff85afcd020b29a8d25e95" exitCode=0
Sep 15 22:40:44.771849 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:44.771444    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" event={"ID":"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca","Type":"ContainerDied","Data":"c48c3ba34829c989333d521e47efca8c4f0f28a144ff85afcd020b29a8d25e95"}
Sep 15 22:40:44.771904 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:44.771873    2584 scope.go:117] "RemoveContainer" containerID="c48c3ba34829c989333d521e47efca8c4f0f28a144ff85afcd020b29a8d25e95"
Sep 15 22:40:44.781083 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:44.779860    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw" event={"ID":"5bda438d-af76-4c2d-8b4e-f39d13c857ea","Type":"ContainerStarted","Data":"fa427d0e2f6b22e4e4be34af6576dd33bbf2c8ecc3e6b81a4c5770d4b6eb6b9a"}
Sep 15 22:40:44.781083 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:44.779869    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:40:44.781083 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:44.779949    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:40:44.781083 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:44.779996    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:40:45.202716 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:45.202673    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:45.202924 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:45.202847    2584 patch_prober.go:28] interesting pod/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 15 22:40:45.202924 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:45.202894    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw" podUID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 15 22:40:45.321205 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:45.321165    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:45.423781 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:45.423732    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:45.689045 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:45.689012    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:45.793062 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:45.793026    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" event={"ID":"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca","Type":"ContainerStarted","Data":"01ab1a33bb2f3a46cd0861275be2253a0be216842816af8ba669de826b2d9a12"}
Sep 15 22:40:45.978735 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:45.978640    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:46.201717 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:46.201672    2584 patch_prober.go:28] interesting pod/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 15 22:40:46.201920 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:46.201747    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw" podUID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 15 22:40:46.306549 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:46.306514    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:46.314500 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:46.314473    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:46.315007 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:46.314979    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 22:40:46.315243 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:46.315214    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 22:40:46.402767 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:46.402723    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:46.630032 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:46.629931    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:47.202575 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:47.202532    2584 patch_prober.go:28] interesting pod/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 15 22:40:47.203080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:47.202603    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw" podUID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 15 22:40:47.316291 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:47.315385    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 22:40:47.316291 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:47.315465    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 22:40:47.363717 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:47.363679    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:47.460795 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:47.460700    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:47.586990 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:47.586950    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:47.694772 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:47.694729    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:48.202720 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:48.202672    2584 patch_prober.go:28] interesting pod/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 15 22:40:48.203379 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:48.203342    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw" podUID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 15 22:40:48.313924 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:48.313888    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 22:40:48.314247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:48.314195    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 22:40:48.331762 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:48.331709    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:48.884121 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:48.884079    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:49.202436 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.202344    2584 patch_prober.go:28] interesting pod/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 15 22:40:49.202436 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.202415    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw" podUID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 15 22:40:49.223633 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.223591    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:49.313289 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.312990    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:49.317289 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.314609    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 22:40:49.317289 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.314667    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 22:40:49.724726 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.724674    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf"]
Sep 15 22:40:49.728280 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.728246    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf"
Sep 15 22:40:49.731138 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.731116    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:40:49.731469 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.731446    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:40:49.740220 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.740161    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf"]
Sep 15 22:40:49.770682 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.770652    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea0c6cc-41c8-422e-b392-053a49a28d4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf\" (UID: \"eea0c6cc-41c8-422e-b392-053a49a28d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf"
Sep 15 22:40:49.871621 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.871524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea0c6cc-41c8-422e-b392-053a49a28d4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf\" (UID: \"eea0c6cc-41c8-422e-b392-053a49a28d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf"
Sep 15 22:40:49.872486 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:49.872451    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea0c6cc-41c8-422e-b392-053a49a28d4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf\" (UID: \"eea0c6cc-41c8-422e-b392-053a49a28d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf"
Sep 15 22:40:50.039871 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:50.039831    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf"
Sep 15 22:40:50.209623 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:50.209593    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:50.210101 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:50.210071    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:50.210690 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:50.210671    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:40:50.301316 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:50.298633    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:50.313964 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:50.313923    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 22:40:50.314121 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:50.313991    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 22:40:50.345323 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:50.345297    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf"]
Sep 15 22:40:50.346760 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:40:50.346733    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea0c6cc_41c8_422e_b392_053a49a28d4d.slice/crio-0b42505fa39416c7712af642919ccbd2862eaa8899572c9fa836a6a528d1a7d3 WatchSource:0}: Error finding container 0b42505fa39416c7712af642919ccbd2862eaa8899572c9fa836a6a528d1a7d3: Status 404 returned error can't find the container with id 0b42505fa39416c7712af642919ccbd2862eaa8899572c9fa836a6a528d1a7d3
Sep 15 22:40:50.842721 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:50.842688    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf" event={"ID":"eea0c6cc-41c8-422e-b392-053a49a28d4d","Type":"ContainerStarted","Data":"0b42505fa39416c7712af642919ccbd2862eaa8899572c9fa836a6a528d1a7d3"}
Sep 15 22:40:51.279150 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:51.279039    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:51.314423 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:51.314380    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 22:40:51.314819 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:51.314455    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 22:40:51.752076 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:51.752046    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"
Sep 15 22:40:51.752254 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:51.752179    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"
Sep 15 22:40:52.002046 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:52.001999    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:52.318220 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:52.318192    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:52.318635 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:52.318512    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:52.319128 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:52.319105    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:40:52.759154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:52.759071    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"
Sep 15 22:40:52.884583 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:52.884544    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:52.984355 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:52.984320    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:53.505544 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.505498    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:53.605069 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.605007    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:53.640640 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.640608    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"]
Sep 15 22:40:53.649430 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.649406    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"
Sep 15 22:40:53.652473 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.652436    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-765e6d-dockercfg-l8mkj\""
Sep 15 22:40:53.652911 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.652738    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-765e6d\""
Sep 15 22:40:53.657528 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.657507    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"]
Sep 15 22:40:53.737329 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.737255    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znsr9\" (UniqueName: \"kubernetes.io/projected/b4d5cd52-9645-421a-a037-613918ea98ed-kube-api-access-znsr9\") pod \"payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5\" (UID: \"b4d5cd52-9645-421a-a037-613918ea98ed\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"
Sep 15 22:40:53.737456 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.737424    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d5cd52-9645-421a-a037-613918ea98ed-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5\" (UID: \"b4d5cd52-9645-421a-a037-613918ea98ed\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"
Sep 15 22:40:53.830667 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.830625    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:53.838759 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.838733    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d5cd52-9645-421a-a037-613918ea98ed-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5\" (UID: \"b4d5cd52-9645-421a-a037-613918ea98ed\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"
Sep 15 22:40:53.838861 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.838814    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znsr9\" (UniqueName: \"kubernetes.io/projected/b4d5cd52-9645-421a-a037-613918ea98ed-kube-api-access-znsr9\") pod \"payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5\" (UID: \"b4d5cd52-9645-421a-a037-613918ea98ed\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"
Sep 15 22:40:53.839438 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.839420    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d5cd52-9645-421a-a037-613918ea98ed-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5\" (UID: \"b4d5cd52-9645-421a-a037-613918ea98ed\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"
Sep 15 22:40:53.848632 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.848605    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znsr9\" (UniqueName: \"kubernetes.io/projected/b4d5cd52-9645-421a-a037-613918ea98ed-kube-api-access-znsr9\") pod \"payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5\" (UID: \"b4d5cd52-9645-421a-a037-613918ea98ed\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"
Sep 15 22:40:53.966585 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.966535    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:53.967366 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:53.967347    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"
Sep 15 22:40:54.073210 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:54.073171    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:54.168109 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:54.168071    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:54.250916 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:54.250889    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"]
Sep 15 22:40:54.252425 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:40:54.252390    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d5cd52_9645_421a_a037_613918ea98ed.slice/crio-aa2c121f403193e7c6fe7782d1bcce0d9cb8a7eb91ccf20eaf0a828e994a1c9a WatchSource:0}: Error finding container aa2c121f403193e7c6fe7782d1bcce0d9cb8a7eb91ccf20eaf0a828e994a1c9a: Status 404 returned error can't find the container with id aa2c121f403193e7c6fe7782d1bcce0d9cb8a7eb91ccf20eaf0a828e994a1c9a
Sep 15 22:40:54.315373 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:54.315344    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:54.873065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:54.872964    2584 generic.go:356] "Generic (PLEG): container finished" podID="eea0c6cc-41c8-422e-b392-053a49a28d4d" containerID="ca525e141e6a8f647b5758ea0d6c4f877dedb1c941fe2e83b0864f256de7aa48" exitCode=0
Sep 15 22:40:54.873065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:54.873017    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf" event={"ID":"eea0c6cc-41c8-422e-b392-053a49a28d4d","Type":"ContainerDied","Data":"ca525e141e6a8f647b5758ea0d6c4f877dedb1c941fe2e83b0864f256de7aa48"}
Sep 15 22:40:54.875022 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:54.874997    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5" event={"ID":"b4d5cd52-9645-421a-a037-613918ea98ed","Type":"ContainerStarted","Data":"c8e0c7f207b3c9fa20ff10ae22c740e62e7619be092f6d3675e18beec8097e33"}
Sep 15 22:40:54.875162 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:54.875037    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5" event={"ID":"b4d5cd52-9645-421a-a037-613918ea98ed","Type":"ContainerStarted","Data":"aa2c121f403193e7c6fe7782d1bcce0d9cb8a7eb91ccf20eaf0a828e994a1c9a"}
Sep 15 22:40:54.875233 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:54.875161    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"
Sep 15 22:40:54.907173 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:54.907128    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5" podStartSLOduration=1.907113481 podStartE2EDuration="1.907113481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:40:54.905375335 +0000 UTC m=+827.524190248" watchObservedRunningTime="2026-09-15 22:40:54.907113481 +0000 UTC m=+827.525928388"
Sep 15 22:40:54.972068 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:54.972036    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:55.779812 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:55.779777    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:56.023878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:56.023850    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf"
Sep 15 22:40:56.161433 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:56.161403    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea0c6cc-41c8-422e-b392-053a49a28d4d-maas-api-service-ca\") pod \"eea0c6cc-41c8-422e-b392-053a49a28d4d\" (UID: \"eea0c6cc-41c8-422e-b392-053a49a28d4d\") "
Sep 15 22:40:56.161774 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:56.161749    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea0c6cc-41c8-422e-b392-053a49a28d4d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eea0c6cc-41c8-422e-b392-053a49a28d4d" (UID: "eea0c6cc-41c8-422e-b392-053a49a28d4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:40:56.263074 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:56.263038    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea0c6cc-41c8-422e-b392-053a49a28d4d-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:40:56.486140 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:56.486018    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:56.887061 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:56.887029    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf"
Sep 15 22:40:56.887061 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:56.887050    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf" event={"ID":"eea0c6cc-41c8-422e-b392-053a49a28d4d","Type":"ContainerDied","Data":"0b42505fa39416c7712af642919ccbd2862eaa8899572c9fa836a6a528d1a7d3"}
Sep 15 22:40:56.887298 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:56.887100    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b42505fa39416c7712af642919ccbd2862eaa8899572c9fa836a6a528d1a7d3"
Sep 15 22:40:57.177424 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:57.177331    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:57.876074 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:57.876026    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:57.977663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:57.977632    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:58.646347 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:58.646315    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:40:59.512739 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:40:59.512696    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:00.400638 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:00.400605    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:00.501109 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:00.501078    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:00.600933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:00.600890    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:00.996533 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:00.996497    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:01.040046 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.040014    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"]
Sep 15 22:41:01.040299 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.040253    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v" podUID="26ba2ee6-ae70-46d7-aded-9d54ed80e11d" containerName="payload-processing" containerID="cri-o://177c40a681eebfafed42886493c3d05686b5f0e9890275dd32b552be224e82c6" gracePeriod=30
Sep 15 22:41:01.046929 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.046793    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"]
Sep 15 22:41:01.047096 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.047007    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf" podUID="d387ab83-e4df-413a-9341-9b314b9a3947" containerName="payload-pre-processing" containerID="cri-o://4a048f49a4cad04bd461ebefea68e6c1e2b097117ce465651b0aa08fa013e215" gracePeriod=30
Sep 15 22:41:01.083446 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.083415    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:01.144748 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.144697    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:01.213361 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.212027    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf"]
Sep 15 22:41:01.221675 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.221419    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33ff1a95-nv2xf"]
Sep 15 22:41:01.222593 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.222538    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:01.278490 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.278460    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:41:01.299797 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.299742    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:01.331982 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.331362    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"
Sep 15 22:41:01.335376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.335349    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"
Sep 15 22:41:01.390163 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.390113    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:01.512358 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.512251    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7bjg\" (UniqueName: \"kubernetes.io/projected/26ba2ee6-ae70-46d7-aded-9d54ed80e11d-kube-api-access-d7bjg\") pod \"26ba2ee6-ae70-46d7-aded-9d54ed80e11d\" (UID: \"26ba2ee6-ae70-46d7-aded-9d54ed80e11d\") "
Sep 15 22:41:01.512358 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.512309    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ba2ee6-ae70-46d7-aded-9d54ed80e11d-plugins-config-volume\") pod \"26ba2ee6-ae70-46d7-aded-9d54ed80e11d\" (UID: \"26ba2ee6-ae70-46d7-aded-9d54ed80e11d\") "
Sep 15 22:41:01.512871 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.512468    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zdmq\" (UniqueName: \"kubernetes.io/projected/d387ab83-e4df-413a-9341-9b314b9a3947-kube-api-access-5zdmq\") pod \"d387ab83-e4df-413a-9341-9b314b9a3947\" (UID: \"d387ab83-e4df-413a-9341-9b314b9a3947\") "
Sep 15 22:41:01.512871 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.512508    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d387ab83-e4df-413a-9341-9b314b9a3947-plugins-config-volume\") pod \"d387ab83-e4df-413a-9341-9b314b9a3947\" (UID: \"d387ab83-e4df-413a-9341-9b314b9a3947\") "
Sep 15 22:41:01.512871 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.512747    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ba2ee6-ae70-46d7-aded-9d54ed80e11d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26ba2ee6-ae70-46d7-aded-9d54ed80e11d" (UID: "26ba2ee6-ae70-46d7-aded-9d54ed80e11d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:41:01.512990 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.512864    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ba2ee6-ae70-46d7-aded-9d54ed80e11d-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:41:01.512990 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.512943    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d387ab83-e4df-413a-9341-9b314b9a3947-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d387ab83-e4df-413a-9341-9b314b9a3947" (UID: "d387ab83-e4df-413a-9341-9b314b9a3947"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:41:01.514653 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.514620    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d387ab83-e4df-413a-9341-9b314b9a3947-kube-api-access-5zdmq" (OuterVolumeSpecName: "kube-api-access-5zdmq") pod "d387ab83-e4df-413a-9341-9b314b9a3947" (UID: "d387ab83-e4df-413a-9341-9b314b9a3947"). InnerVolumeSpecName "kube-api-access-5zdmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:41:01.514653 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.514627    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ba2ee6-ae70-46d7-aded-9d54ed80e11d-kube-api-access-d7bjg" (OuterVolumeSpecName: "kube-api-access-d7bjg") pod "26ba2ee6-ae70-46d7-aded-9d54ed80e11d" (UID: "26ba2ee6-ae70-46d7-aded-9d54ed80e11d"). InnerVolumeSpecName "kube-api-access-d7bjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:41:01.613713 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.613680    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zdmq\" (UniqueName: \"kubernetes.io/projected/d387ab83-e4df-413a-9341-9b314b9a3947-kube-api-access-5zdmq\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:41:01.613713 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.613707    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d387ab83-e4df-413a-9341-9b314b9a3947-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:41:01.613713 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.613718    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7bjg\" (UniqueName: \"kubernetes.io/projected/26ba2ee6-ae70-46d7-aded-9d54ed80e11d-kube-api-access-d7bjg\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:41:01.836117 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.836083    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:01.899722 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.899687    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eea0c6cc-41c8-422e-b392-053a49a28d4d" path="/var/lib/kubelet/pods/eea0c6cc-41c8-422e-b392-053a49a28d4d/volumes"
Sep 15 22:41:01.914313 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.913824    2584 generic.go:356] "Generic (PLEG): container finished" podID="d387ab83-e4df-413a-9341-9b314b9a3947" containerID="4a048f49a4cad04bd461ebefea68e6c1e2b097117ce465651b0aa08fa013e215" exitCode=0
Sep 15 22:41:01.914313 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.913901    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf" event={"ID":"d387ab83-e4df-413a-9341-9b314b9a3947","Type":"ContainerDied","Data":"4a048f49a4cad04bd461ebefea68e6c1e2b097117ce465651b0aa08fa013e215"}
Sep 15 22:41:01.914313 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.913982    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf" event={"ID":"d387ab83-e4df-413a-9341-9b314b9a3947","Type":"ContainerDied","Data":"e533f5c8e6c3a7b5b662f34b91ce2328011d950dee8d54ec9a129a8534d99230"}
Sep 15 22:41:01.914313 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.914029    2584 scope.go:117] "RemoveContainer" containerID="4a048f49a4cad04bd461ebefea68e6c1e2b097117ce465651b0aa08fa013e215"
Sep 15 22:41:01.914313 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.914203    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"
Sep 15 22:41:01.920835 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.920804    2584 generic.go:356] "Generic (PLEG): container finished" podID="26ba2ee6-ae70-46d7-aded-9d54ed80e11d" containerID="177c40a681eebfafed42886493c3d05686b5f0e9890275dd32b552be224e82c6" exitCode=0
Sep 15 22:41:01.920953 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.920869    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v" event={"ID":"26ba2ee6-ae70-46d7-aded-9d54ed80e11d","Type":"ContainerDied","Data":"177c40a681eebfafed42886493c3d05686b5f0e9890275dd32b552be224e82c6"}
Sep 15 22:41:01.920953 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.920915    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v" event={"ID":"26ba2ee6-ae70-46d7-aded-9d54ed80e11d","Type":"ContainerDied","Data":"423ed18f784359c311417f79e1358a003032136b475a4be15bf9e8eb6fbece45"}
Sep 15 22:41:01.920953 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.920932    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"
Sep 15 22:41:01.926814 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.926783    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:01.930732 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.930715    2584 scope.go:117] "RemoveContainer" containerID="4a048f49a4cad04bd461ebefea68e6c1e2b097117ce465651b0aa08fa013e215"
Sep 15 22:41:01.931033 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:41:01.931008    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a048f49a4cad04bd461ebefea68e6c1e2b097117ce465651b0aa08fa013e215\": container with ID starting with 4a048f49a4cad04bd461ebefea68e6c1e2b097117ce465651b0aa08fa013e215 not found: ID does not exist" containerID="4a048f49a4cad04bd461ebefea68e6c1e2b097117ce465651b0aa08fa013e215"
Sep 15 22:41:01.931119 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.931043    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a048f49a4cad04bd461ebefea68e6c1e2b097117ce465651b0aa08fa013e215"} err="failed to get container status \"4a048f49a4cad04bd461ebefea68e6c1e2b097117ce465651b0aa08fa013e215\": rpc error: code = NotFound desc = could not find container \"4a048f49a4cad04bd461ebefea68e6c1e2b097117ce465651b0aa08fa013e215\": container with ID starting with 4a048f49a4cad04bd461ebefea68e6c1e2b097117ce465651b0aa08fa013e215 not found: ID does not exist"
Sep 15 22:41:01.931119 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.931060    2584 scope.go:117] "RemoveContainer" containerID="177c40a681eebfafed42886493c3d05686b5f0e9890275dd32b552be224e82c6"
Sep 15 22:41:01.936070 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.936041    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"]
Sep 15 22:41:01.938195 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.938174    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33ff1a95-84dd4788db-8d6rf"]
Sep 15 22:41:01.940621 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.940605    2584 scope.go:117] "RemoveContainer" containerID="177c40a681eebfafed42886493c3d05686b5f0e9890275dd32b552be224e82c6"
Sep 15 22:41:01.940912 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:41:01.940893    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177c40a681eebfafed42886493c3d05686b5f0e9890275dd32b552be224e82c6\": container with ID starting with 177c40a681eebfafed42886493c3d05686b5f0e9890275dd32b552be224e82c6 not found: ID does not exist" containerID="177c40a681eebfafed42886493c3d05686b5f0e9890275dd32b552be224e82c6"
Sep 15 22:41:01.940957 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.940924    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177c40a681eebfafed42886493c3d05686b5f0e9890275dd32b552be224e82c6"} err="failed to get container status \"177c40a681eebfafed42886493c3d05686b5f0e9890275dd32b552be224e82c6\": rpc error: code = NotFound desc = could not find container \"177c40a681eebfafed42886493c3d05686b5f0e9890275dd32b552be224e82c6\": container with ID starting with 177c40a681eebfafed42886493c3d05686b5f0e9890275dd32b552be224e82c6 not found: ID does not exist"
Sep 15 22:41:01.948536 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.948513    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"]
Sep 15 22:41:01.952757 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:01.952739    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33ff1a95-cb6d95d4c-d4v2v"]
Sep 15 22:41:02.034118 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:02.034083    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:02.349590 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:02.349553    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:02.446221 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:02.446177    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:02.570054 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:02.570012    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:03.328106 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:03.328073    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:03.899126 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:03.899090    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26ba2ee6-ae70-46d7-aded-9d54ed80e11d" path="/var/lib/kubelet/pods/26ba2ee6-ae70-46d7-aded-9d54ed80e11d/volumes"
Sep 15 22:41:03.899605 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:03.899587    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d387ab83-e4df-413a-9341-9b314b9a3947" path="/var/lib/kubelet/pods/d387ab83-e4df-413a-9341-9b314b9a3947/volumes"
Sep 15 22:41:04.153561 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:04.153473    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:04.718533 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:04.718500    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:04.817969 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:04.817935    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:04.875259 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:04.875225    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:05.078540 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:05.078499    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:05.868927 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:05.868882    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:05.883163 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:05.883138    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"
Sep 15 22:41:06.775780 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:06.775739    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:07.699399 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:07.699363    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:07.760732 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:07.760695    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:08.160831 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.160797    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:08.262006 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.261943    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:08.308508 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.308483    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"]
Sep 15 22:41:08.308916 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.308904    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ba2ee6-ae70-46d7-aded-9d54ed80e11d" containerName="payload-processing"
Sep 15 22:41:08.308960 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.308918    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ba2ee6-ae70-46d7-aded-9d54ed80e11d" containerName="payload-processing"
Sep 15 22:41:08.308960 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.308928    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea0c6cc-41c8-422e-b392-053a49a28d4d" containerName="revoke-keys"
Sep 15 22:41:08.308960 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.308934    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea0c6cc-41c8-422e-b392-053a49a28d4d" containerName="revoke-keys"
Sep 15 22:41:08.308960 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.308948    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d387ab83-e4df-413a-9341-9b314b9a3947" containerName="payload-pre-processing"
Sep 15 22:41:08.308960 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.308954    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d387ab83-e4df-413a-9341-9b314b9a3947" containerName="payload-pre-processing"
Sep 15 22:41:08.309110 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.309013    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea0c6cc-41c8-422e-b392-053a49a28d4d" containerName="revoke-keys"
Sep 15 22:41:08.309110 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.309024    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ba2ee6-ae70-46d7-aded-9d54ed80e11d" containerName="payload-processing"
Sep 15 22:41:08.309110 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.309031    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d387ab83-e4df-413a-9341-9b314b9a3947" containerName="payload-pre-processing"
Sep 15 22:41:08.313488 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.313468    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"
Sep 15 22:41:08.316160 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.316142    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-36d266fd\""
Sep 15 22:41:08.316257 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.316143    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-36d266fd-dockercfg-lptfz\""
Sep 15 22:41:08.321887 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.321853    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"]
Sep 15 22:41:08.376333 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.376284    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fddbb29-e0db-437c-b576-f28e12547f10-plugins-config-volume\") pod \"payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q\" (UID: \"5fddbb29-e0db-437c-b576-f28e12547f10\") " pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"
Sep 15 22:41:08.376458 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.376410    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmk7p\" (UniqueName: \"kubernetes.io/projected/5fddbb29-e0db-437c-b576-f28e12547f10-kube-api-access-bmk7p\") pod \"payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q\" (UID: \"5fddbb29-e0db-437c-b576-f28e12547f10\") " pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"
Sep 15 22:41:08.477510 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.477429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmk7p\" (UniqueName: \"kubernetes.io/projected/5fddbb29-e0db-437c-b576-f28e12547f10-kube-api-access-bmk7p\") pod \"payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q\" (UID: \"5fddbb29-e0db-437c-b576-f28e12547f10\") " pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"
Sep 15 22:41:08.477510 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.477491    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fddbb29-e0db-437c-b576-f28e12547f10-plugins-config-volume\") pod \"payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q\" (UID: \"5fddbb29-e0db-437c-b576-f28e12547f10\") " pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"
Sep 15 22:41:08.478248 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.478227    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fddbb29-e0db-437c-b576-f28e12547f10-plugins-config-volume\") pod \"payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q\" (UID: \"5fddbb29-e0db-437c-b576-f28e12547f10\") " pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"
Sep 15 22:41:08.485799 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.485776    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmk7p\" (UniqueName: \"kubernetes.io/projected/5fddbb29-e0db-437c-b576-f28e12547f10-kube-api-access-bmk7p\") pod \"payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q\" (UID: \"5fddbb29-e0db-437c-b576-f28e12547f10\") " pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"
Sep 15 22:41:08.625003 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.624956    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"
Sep 15 22:41:08.669656 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.669627    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:08.901006 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.900975    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"]
Sep 15 22:41:08.904345 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:41:08.904315    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fddbb29_e0db_437c_b576_f28e12547f10.slice/crio-402c29efc4817146e19500e48789197e91f5d39c58687f8b5d1cbd8cf6767b74 WatchSource:0}: Error finding container 402c29efc4817146e19500e48789197e91f5d39c58687f8b5d1cbd8cf6767b74: Status 404 returned error can't find the container with id 402c29efc4817146e19500e48789197e91f5d39c58687f8b5d1cbd8cf6767b74
Sep 15 22:41:08.922872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.922847    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:08.964481 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:08.964454    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q" event={"ID":"5fddbb29-e0db-437c-b576-f28e12547f10","Type":"ContainerStarted","Data":"402c29efc4817146e19500e48789197e91f5d39c58687f8b5d1cbd8cf6767b74"}
Sep 15 22:41:09.567061 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:09.566997    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:09.971799 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:09.971701    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q" event={"ID":"5fddbb29-e0db-437c-b576-f28e12547f10","Type":"ContainerStarted","Data":"2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca"}
Sep 15 22:41:09.971799 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:09.971767    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"
Sep 15 22:41:09.988720 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:09.988676    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q" podStartSLOduration=1.9886618870000001 podStartE2EDuration="1.988661887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:41:09.987527401 +0000 UTC m=+842.606342310" watchObservedRunningTime="2026-09-15 22:41:09.988661887 +0000 UTC m=+842.607476794"
Sep 15 22:41:10.359081 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:10.359036    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:11.134250 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:11.134217    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:11.832766 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:11.832725    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:12.549783 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:12.549737    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:13.477755 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:13.477700    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:14.183478 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:14.183434    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:20.569428 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:20.569395    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:41:20.980511 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:20.980415    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"
Sep 15 22:41:28.621377 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:28.621335    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:30.071139 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:30.071101    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:30.879636 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:30.879598    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:31.695319 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:31.695279    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:32.471782 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:32.471732    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:33.182654 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:33.182619    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:33.892030 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:33.891994    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:34.356849 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:34.356817    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:34.586533 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:34.586497    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:34.819804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:34.819771    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv"]
Sep 15 22:41:34.823449 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:34.823426    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv"
Sep 15 22:41:34.826048 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:34.826026    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:41:34.826146 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:34.826026    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:41:34.830733 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:34.830710    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv"]
Sep 15 22:41:34.918634 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:34.918598    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d2d997-2a9f-4252-9ec6-91c9cb92642b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv\" (UID: \"00d2d997-2a9f-4252-9ec6-91c9cb92642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv"
Sep 15 22:41:35.019855 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:35.019817    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d2d997-2a9f-4252-9ec6-91c9cb92642b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv\" (UID: \"00d2d997-2a9f-4252-9ec6-91c9cb92642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv"
Sep 15 22:41:35.020696 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:35.020667    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d2d997-2a9f-4252-9ec6-91c9cb92642b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv\" (UID: \"00d2d997-2a9f-4252-9ec6-91c9cb92642b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv"
Sep 15 22:41:35.135958 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:35.135884    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv"
Sep 15 22:41:35.389330 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:35.389203    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:35.407331 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:35.407301    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv"]
Sep 15 22:41:35.409082 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:41:35.409054    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d2d997_2a9f_4252_9ec6_91c9cb92642b.slice/crio-afbab6d6bd5623663280ba5ba9b90321c5256d06f0bfd60f83965ba22ae65524 WatchSource:0}: Error finding container afbab6d6bd5623663280ba5ba9b90321c5256d06f0bfd60f83965ba22ae65524: Status 404 returned error can't find the container with id afbab6d6bd5623663280ba5ba9b90321c5256d06f0bfd60f83965ba22ae65524
Sep 15 22:41:36.116975 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:36.116930    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv" event={"ID":"00d2d997-2a9f-4252-9ec6-91c9cb92642b","Type":"ContainerStarted","Data":"638acfcc818a7b936f0635c2ddfef27876e81e521c8f62435ca0280d6acaa793"}
Sep 15 22:41:36.117137 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:36.116990    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv" event={"ID":"00d2d997-2a9f-4252-9ec6-91c9cb92642b","Type":"ContainerStarted","Data":"afbab6d6bd5623663280ba5ba9b90321c5256d06f0bfd60f83965ba22ae65524"}
Sep 15 22:41:36.134476 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:36.134428    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv" podStartSLOduration=2.134413379 podStartE2EDuration="2.134413379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:41:36.132814914 +0000 UTC m=+868.751629822" watchObservedRunningTime="2026-09-15 22:41:36.134413379 +0000 UTC m=+868.753228286"
Sep 15 22:41:36.282753 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:36.282720    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:37.041570 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:37.041531    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:37.123445 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:37.123409    2584 generic.go:356] "Generic (PLEG): container finished" podID="00d2d997-2a9f-4252-9ec6-91c9cb92642b" containerID="638acfcc818a7b936f0635c2ddfef27876e81e521c8f62435ca0280d6acaa793" exitCode=0
Sep 15 22:41:37.123620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:37.123491    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv" event={"ID":"00d2d997-2a9f-4252-9ec6-91c9cb92642b","Type":"ContainerDied","Data":"638acfcc818a7b936f0635c2ddfef27876e81e521c8f62435ca0280d6acaa793"}
Sep 15 22:41:37.852025 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:37.851983    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:38.287012 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:38.286985    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv"
Sep 15 22:41:38.453078 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:38.452992    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d2d997-2a9f-4252-9ec6-91c9cb92642b-maas-api-service-ca\") pod \"00d2d997-2a9f-4252-9ec6-91c9cb92642b\" (UID: \"00d2d997-2a9f-4252-9ec6-91c9cb92642b\") "
Sep 15 22:41:38.453421 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:38.453394    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d2d997-2a9f-4252-9ec6-91c9cb92642b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00d2d997-2a9f-4252-9ec6-91c9cb92642b" (UID: "00d2d997-2a9f-4252-9ec6-91c9cb92642b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:41:38.554112 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:38.554074    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d2d997-2a9f-4252-9ec6-91c9cb92642b-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:41:38.584122 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:38.584090    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:39.135990 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:39.135959    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv"
Sep 15 22:41:39.135990 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:39.135964    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv" event={"ID":"00d2d997-2a9f-4252-9ec6-91c9cb92642b","Type":"ContainerDied","Data":"afbab6d6bd5623663280ba5ba9b90321c5256d06f0bfd60f83965ba22ae65524"}
Sep 15 22:41:39.136192 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:39.136016    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afbab6d6bd5623663280ba5ba9b90321c5256d06f0bfd60f83965ba22ae65524"
Sep 15 22:41:39.269137 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:39.269101    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:39.667695 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:39.667666    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:41:39.989714 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:39.989626    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:46.371768 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:46.371726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:46.406718 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:46.406660    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"]
Sep 15 22:41:46.407355 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:46.407299    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q" podUID="5fddbb29-e0db-437c-b576-f28e12547f10" containerName="payload-processing" containerID="cri-o://2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca" gracePeriod=30
Sep 15 22:41:46.456541 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:41:46.456497    2584 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fddbb29_e0db_437c_b576_f28e12547f10.slice/crio-conmon-2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fddbb29_e0db_437c_b576_f28e12547f10.slice/crio-2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca.scope\": RecentStats: unable to find data in memory cache]"
Sep 15 22:41:46.560385 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:46.560329    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv"]
Sep 15 22:41:46.562436 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:46.562410    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-36d266fd-xjnpv"]
Sep 15 22:41:46.706174 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:46.702658    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"
Sep 15 22:41:46.826808 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:46.826772    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fddbb29-e0db-437c-b576-f28e12547f10-plugins-config-volume\") pod \"5fddbb29-e0db-437c-b576-f28e12547f10\" (UID: \"5fddbb29-e0db-437c-b576-f28e12547f10\") "
Sep 15 22:41:46.827064 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:46.826913    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmk7p\" (UniqueName: \"kubernetes.io/projected/5fddbb29-e0db-437c-b576-f28e12547f10-kube-api-access-bmk7p\") pod \"5fddbb29-e0db-437c-b576-f28e12547f10\" (UID: \"5fddbb29-e0db-437c-b576-f28e12547f10\") "
Sep 15 22:41:46.827166 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:46.827138    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fddbb29-e0db-437c-b576-f28e12547f10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5fddbb29-e0db-437c-b576-f28e12547f10" (UID: "5fddbb29-e0db-437c-b576-f28e12547f10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:41:46.827365 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:46.827313    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fddbb29-e0db-437c-b576-f28e12547f10-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:41:46.829532 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:46.829506    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fddbb29-e0db-437c-b576-f28e12547f10-kube-api-access-bmk7p" (OuterVolumeSpecName: "kube-api-access-bmk7p") pod "5fddbb29-e0db-437c-b576-f28e12547f10" (UID: "5fddbb29-e0db-437c-b576-f28e12547f10"). InnerVolumeSpecName "kube-api-access-bmk7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:41:46.927859 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:46.927769    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmk7p\" (UniqueName: \"kubernetes.io/projected/5fddbb29-e0db-437c-b576-f28e12547f10-kube-api-access-bmk7p\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:41:47.178459 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:47.178369    2584 generic.go:356] "Generic (PLEG): container finished" podID="5fddbb29-e0db-437c-b576-f28e12547f10" containerID="2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca" exitCode=0
Sep 15 22:41:47.178459 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:47.178430    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"
Sep 15 22:41:47.178635 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:47.178458    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q" event={"ID":"5fddbb29-e0db-437c-b576-f28e12547f10","Type":"ContainerDied","Data":"2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca"}
Sep 15 22:41:47.178635 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:47.178515    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q" event={"ID":"5fddbb29-e0db-437c-b576-f28e12547f10","Type":"ContainerDied","Data":"402c29efc4817146e19500e48789197e91f5d39c58687f8b5d1cbd8cf6767b74"}
Sep 15 22:41:47.178635 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:47.178537    2584 scope.go:117] "RemoveContainer" containerID="2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca"
Sep 15 22:41:47.187922 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:47.187904    2584 scope.go:117] "RemoveContainer" containerID="2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca"
Sep 15 22:41:47.188205 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:41:47.188185    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca\": container with ID starting with 2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca not found: ID does not exist" containerID="2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca"
Sep 15 22:41:47.188253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:47.188215    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca"} err="failed to get container status \"2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca\": rpc error: code = NotFound desc = could not find container \"2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca\": container with ID starting with 2e7894541a0c349e3809e7125cbc0a1e224425784937d1118f46b50df0a0dcca not found: ID does not exist"
Sep 15 22:41:47.200326 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:47.200299    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"]
Sep 15 22:41:47.203911 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:47.203888    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-36d266fd-85c6b89f98-v2h9q"]
Sep 15 22:41:47.885868 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:47.885835    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:47.900368 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:47.900335    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d2d997-2a9f-4252-9ec6-91c9cb92642b" path="/var/lib/kubelet/pods/00d2d997-2a9f-4252-9ec6-91c9cb92642b/volumes"
Sep 15 22:41:47.900823 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:47.900803    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fddbb29-e0db-437c-b576-f28e12547f10" path="/var/lib/kubelet/pods/5fddbb29-e0db-437c-b576-f28e12547f10/volumes"
Sep 15 22:41:48.036677 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:48.036627    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:48.734866 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:48.734789    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:49.484486 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:49.484450    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:49.923871 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:49.923820    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:50.024970 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:50.024934    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:50.318326 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:50.318289    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:51.035372 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:51.035339    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:51.975456 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:51.975419    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:52.032333 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.032295    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:52.559255 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.559216    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:52.650418 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.650385    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"]
Sep 15 22:41:52.650821 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.650807    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fddbb29-e0db-437c-b576-f28e12547f10" containerName="payload-processing"
Sep 15 22:41:52.650867 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.650822    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fddbb29-e0db-437c-b576-f28e12547f10" containerName="payload-processing"
Sep 15 22:41:52.650867 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.650836    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d2d997-2a9f-4252-9ec6-91c9cb92642b" containerName="revoke-keys"
Sep 15 22:41:52.650867 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.650842    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d2d997-2a9f-4252-9ec6-91c9cb92642b" containerName="revoke-keys"
Sep 15 22:41:52.650965 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.650902    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fddbb29-e0db-437c-b576-f28e12547f10" containerName="payload-processing"
Sep 15 22:41:52.650965 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.650909    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d2d997-2a9f-4252-9ec6-91c9cb92642b" containerName="revoke-keys"
Sep 15 22:41:52.652946 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.652912    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:52.655411 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.655388    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"
Sep 15 22:41:52.658052 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.658031    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-93550aef-dockercfg-n7v2c\""
Sep 15 22:41:52.658215 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.658187    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-93550aef\""
Sep 15 22:41:52.664314 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.664292    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"]
Sep 15 22:41:52.683777 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.683743    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"]
Sep 15 22:41:52.687002 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.686980    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mjrb\" (UniqueName: \"kubernetes.io/projected/56df1516-45a0-4aab-9dbd-b20586a3cea5-kube-api-access-2mjrb\") pod \"payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh\" (UID: \"56df1516-45a0-4aab-9dbd-b20586a3cea5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"
Sep 15 22:41:52.687082 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.687039    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56df1516-45a0-4aab-9dbd-b20586a3cea5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh\" (UID: \"56df1516-45a0-4aab-9dbd-b20586a3cea5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"
Sep 15 22:41:52.687385 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.687369    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"
Sep 15 22:41:52.701705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.701681    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"]
Sep 15 22:41:52.788073 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.788040    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56df1516-45a0-4aab-9dbd-b20586a3cea5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh\" (UID: \"56df1516-45a0-4aab-9dbd-b20586a3cea5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"
Sep 15 22:41:52.788243 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.788091    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6a1e35-ef8c-4d6f-a845-9caeba54d5e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm\" (UID: \"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6\") " pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"
Sep 15 22:41:52.788243 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.788125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gxh2\" (UniqueName: \"kubernetes.io/projected/de6a1e35-ef8c-4d6f-a845-9caeba54d5e6-kube-api-access-4gxh2\") pod \"payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm\" (UID: \"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6\") " pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"
Sep 15 22:41:52.788243 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.788213    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mjrb\" (UniqueName: \"kubernetes.io/projected/56df1516-45a0-4aab-9dbd-b20586a3cea5-kube-api-access-2mjrb\") pod \"payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh\" (UID: \"56df1516-45a0-4aab-9dbd-b20586a3cea5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"
Sep 15 22:41:52.788774 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.788755    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56df1516-45a0-4aab-9dbd-b20586a3cea5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh\" (UID: \"56df1516-45a0-4aab-9dbd-b20586a3cea5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"
Sep 15 22:41:52.802045 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.802021    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mjrb\" (UniqueName: \"kubernetes.io/projected/56df1516-45a0-4aab-9dbd-b20586a3cea5-kube-api-access-2mjrb\") pod \"payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh\" (UID: \"56df1516-45a0-4aab-9dbd-b20586a3cea5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"
Sep 15 22:41:52.889720 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.889651    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6a1e35-ef8c-4d6f-a845-9caeba54d5e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm\" (UID: \"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6\") " pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"
Sep 15 22:41:52.889720 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.889697    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gxh2\" (UniqueName: \"kubernetes.io/projected/de6a1e35-ef8c-4d6f-a845-9caeba54d5e6-kube-api-access-4gxh2\") pod \"payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm\" (UID: \"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6\") " pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"
Sep 15 22:41:52.890391 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.890356    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6a1e35-ef8c-4d6f-a845-9caeba54d5e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm\" (UID: \"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6\") " pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"
Sep 15 22:41:52.901241 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.901214    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gxh2\" (UniqueName: \"kubernetes.io/projected/de6a1e35-ef8c-4d6f-a845-9caeba54d5e6-kube-api-access-4gxh2\") pod \"payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm\" (UID: \"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6\") " pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"
Sep 15 22:41:52.966517 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.966493    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"
Sep 15 22:41:52.996808 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:52.996781    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"
Sep 15 22:41:53.055626 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:53.055515    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:53.243151 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:53.243058    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"]
Sep 15 22:41:53.246306 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:41:53.246274    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56df1516_45a0_4aab_9dbd_b20586a3cea5.slice/crio-a8350713f3b029144f6b3d11e1b0a6c68a938cc27a274e8fed4c5cc749d2b3f4 WatchSource:0}: Error finding container a8350713f3b029144f6b3d11e1b0a6c68a938cc27a274e8fed4c5cc749d2b3f4: Status 404 returned error can't find the container with id a8350713f3b029144f6b3d11e1b0a6c68a938cc27a274e8fed4c5cc749d2b3f4
Sep 15 22:41:53.279208 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:53.279182    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"]
Sep 15 22:41:53.285775 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:41:53.284538    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6a1e35_ef8c_4d6f_a845_9caeba54d5e6.slice/crio-b780be7c45562beef7f93abf1f1783b0ddd1bcd8bc9ffebbeeaf34df2aef3e3c WatchSource:0}: Error finding container b780be7c45562beef7f93abf1f1783b0ddd1bcd8bc9ffebbeeaf34df2aef3e3c: Status 404 returned error can't find the container with id b780be7c45562beef7f93abf1f1783b0ddd1bcd8bc9ffebbeeaf34df2aef3e3c
Sep 15 22:41:53.464399 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:53.464368    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:41:53.769727 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:53.769689    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:54.217090 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:54.217052    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm" event={"ID":"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6","Type":"ContainerStarted","Data":"e16cbfd8aa44f2a0bffa298286c6e47e01e6da00a247fc093f57cb8e45b28912"}
Sep 15 22:41:54.217316 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:54.217119    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm" event={"ID":"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6","Type":"ContainerStarted","Data":"b780be7c45562beef7f93abf1f1783b0ddd1bcd8bc9ffebbeeaf34df2aef3e3c"}
Sep 15 22:41:54.217316 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:54.217151    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"
Sep 15 22:41:54.218772 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:54.218745    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh" event={"ID":"56df1516-45a0-4aab-9dbd-b20586a3cea5","Type":"ContainerStarted","Data":"04c63b5dc50b55c7826ea09a9a18f2826ffd7c472189dda81c1de69f5af9cae0"}
Sep 15 22:41:54.218872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:54.218796    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh" event={"ID":"56df1516-45a0-4aab-9dbd-b20586a3cea5","Type":"ContainerStarted","Data":"a8350713f3b029144f6b3d11e1b0a6c68a938cc27a274e8fed4c5cc749d2b3f4"}
Sep 15 22:41:54.218872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:54.218836    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"
Sep 15 22:41:54.235977 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:54.235926    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm" podStartSLOduration=2.2359080000000002 podStartE2EDuration="2.235908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:41:54.233860423 +0000 UTC m=+886.852675332" watchObservedRunningTime="2026-09-15 22:41:54.235908 +0000 UTC m=+886.854722911"
Sep 15 22:41:54.250845 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:54.250801    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh" podStartSLOduration=2.250786146 podStartE2EDuration="2.250786146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:41:54.249326501 +0000 UTC m=+886.868141409" watchObservedRunningTime="2026-09-15 22:41:54.250786146 +0000 UTC m=+886.869601055"
Sep 15 22:41:54.643465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:54.643432    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:41:55.325586 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:41:55.325549    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:00.980534 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:00.980430    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:05.225113 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:05.225062    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"
Sep 15 22:42:05.225518 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:05.225249    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"
Sep 15 22:42:07.870844 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:07.870815    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:42:07.871393 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:07.871119    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:42:09.365980 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:09.365946    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:42:14.255149 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:14.255111    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:14.937685 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:14.937646    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:15.606604 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:15.606563    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4"]
Sep 15 22:42:15.611846 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:15.611821    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4"
Sep 15 22:42:15.614688 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:15.614657    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:42:15.614811 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:15.614694    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:42:15.629287 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:15.621732    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4"]
Sep 15 22:42:15.646660 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:15.646622    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:15.708066 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:15.708033    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d707bc00-e569-4930-b1d0-b797d72fb41e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-93550aef-fd5l4\" (UID: \"d707bc00-e569-4930-b1d0-b797d72fb41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4"
Sep 15 22:42:15.809196 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:15.809159    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d707bc00-e569-4930-b1d0-b797d72fb41e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-93550aef-fd5l4\" (UID: \"d707bc00-e569-4930-b1d0-b797d72fb41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4"
Sep 15 22:42:15.809846 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:15.809817    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d707bc00-e569-4930-b1d0-b797d72fb41e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-93550aef-fd5l4\" (UID: \"d707bc00-e569-4930-b1d0-b797d72fb41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4"
Sep 15 22:42:15.929619 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:15.929551    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4"
Sep 15 22:42:16.199357 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:16.199328    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4"]
Sep 15 22:42:16.201247 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:42:16.201215    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd707bc00_e569_4930_b1d0_b797d72fb41e.slice/crio-5a43d3daa93fdf7130515dd45f9d327f9076e12d9c04ca886d644a76da783238 WatchSource:0}: Error finding container 5a43d3daa93fdf7130515dd45f9d327f9076e12d9c04ca886d644a76da783238: Status 404 returned error can't find the container with id 5a43d3daa93fdf7130515dd45f9d327f9076e12d9c04ca886d644a76da783238
Sep 15 22:42:16.339337 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:16.339298    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4" event={"ID":"d707bc00-e569-4930-b1d0-b797d72fb41e","Type":"ContainerStarted","Data":"5a43d3daa93fdf7130515dd45f9d327f9076e12d9c04ca886d644a76da783238"}
Sep 15 22:42:16.463439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:16.463353    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:17.269555 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:17.269457    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:17.345752 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:17.345723    2584 generic.go:356] "Generic (PLEG): container finished" podID="d707bc00-e569-4930-b1d0-b797d72fb41e" containerID="eaa516b835d6225eec07d6ef527df49fd70c735ba2de8a3ad3f18cb777dde924" exitCode=0
Sep 15 22:42:17.345925 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:17.345817    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4" event={"ID":"d707bc00-e569-4930-b1d0-b797d72fb41e","Type":"ContainerDied","Data":"eaa516b835d6225eec07d6ef527df49fd70c735ba2de8a3ad3f18cb777dde924"}
Sep 15 22:42:18.050806 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:18.050770    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:18.492412 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:18.492389    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4"
Sep 15 22:42:18.634566 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:18.634493    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d707bc00-e569-4930-b1d0-b797d72fb41e-maas-api-service-ca\") pod \"d707bc00-e569-4930-b1d0-b797d72fb41e\" (UID: \"d707bc00-e569-4930-b1d0-b797d72fb41e\") "
Sep 15 22:42:18.634839 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:18.634817    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d707bc00-e569-4930-b1d0-b797d72fb41e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d707bc00-e569-4930-b1d0-b797d72fb41e" (UID: "d707bc00-e569-4930-b1d0-b797d72fb41e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:42:18.735775 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:18.735722    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d707bc00-e569-4930-b1d0-b797d72fb41e-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:42:18.872324 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:18.872288    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:19.358096 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:19.358066    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4"
Sep 15 22:42:19.358096 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:19.358095    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4" event={"ID":"d707bc00-e569-4930-b1d0-b797d72fb41e","Type":"ContainerDied","Data":"5a43d3daa93fdf7130515dd45f9d327f9076e12d9c04ca886d644a76da783238"}
Sep 15 22:42:19.358336 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:19.358146    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a43d3daa93fdf7130515dd45f9d327f9076e12d9c04ca886d644a76da783238"
Sep 15 22:42:19.653963 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:19.653881    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:20.357103 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:20.357064    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:21.178741 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:21.178702    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:21.763431 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:21.763389    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:21.786255 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:21.786222    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"]
Sep 15 22:42:21.787030 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:21.786994    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm" podUID="de6a1e35-ef8c-4d6f-a845-9caeba54d5e6" containerName="payload-processing" containerID="cri-o://e16cbfd8aa44f2a0bffa298286c6e47e01e6da00a247fc093f57cb8e45b28912" gracePeriod=30
Sep 15 22:42:21.792706 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:21.792683    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"]
Sep 15 22:42:21.793146 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:21.793076    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh" podUID="56df1516-45a0-4aab-9dbd-b20586a3cea5" containerName="payload-pre-processing" containerID="cri-o://04c63b5dc50b55c7826ea09a9a18f2826ffd7c472189dda81c1de69f5af9cae0" gracePeriod=30
Sep 15 22:42:21.828422 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:21.828388    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:21.889327 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:21.889260    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:21.960319 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:21.959743    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4"]
Sep 15 22:42:21.960424 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:21.960387    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:21.965036 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:21.963975    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-93550aef-fd5l4"]
Sep 15 22:42:22.029395 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.029297    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:22.057792 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.057766    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"
Sep 15 22:42:22.095065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.095033    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"
Sep 15 22:42:22.171011 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.170981    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mjrb\" (UniqueName: \"kubernetes.io/projected/56df1516-45a0-4aab-9dbd-b20586a3cea5-kube-api-access-2mjrb\") pod \"56df1516-45a0-4aab-9dbd-b20586a3cea5\" (UID: \"56df1516-45a0-4aab-9dbd-b20586a3cea5\") "
Sep 15 22:42:22.171157 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.171027    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gxh2\" (UniqueName: \"kubernetes.io/projected/de6a1e35-ef8c-4d6f-a845-9caeba54d5e6-kube-api-access-4gxh2\") pod \"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6\" (UID: \"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6\") "
Sep 15 22:42:22.171157 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.171082    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56df1516-45a0-4aab-9dbd-b20586a3cea5-plugins-config-volume\") pod \"56df1516-45a0-4aab-9dbd-b20586a3cea5\" (UID: \"56df1516-45a0-4aab-9dbd-b20586a3cea5\") "
Sep 15 22:42:22.171157 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.171111    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6a1e35-ef8c-4d6f-a845-9caeba54d5e6-plugins-config-volume\") pod \"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6\" (UID: \"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6\") "
Sep 15 22:42:22.171476 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.171452    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56df1516-45a0-4aab-9dbd-b20586a3cea5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56df1516-45a0-4aab-9dbd-b20586a3cea5" (UID: "56df1516-45a0-4aab-9dbd-b20586a3cea5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:42:22.171563 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.171533    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de6a1e35-ef8c-4d6f-a845-9caeba54d5e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de6a1e35-ef8c-4d6f-a845-9caeba54d5e6" (UID: "de6a1e35-ef8c-4d6f-a845-9caeba54d5e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:42:22.173185 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.173154    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6a1e35-ef8c-4d6f-a845-9caeba54d5e6-kube-api-access-4gxh2" (OuterVolumeSpecName: "kube-api-access-4gxh2") pod "de6a1e35-ef8c-4d6f-a845-9caeba54d5e6" (UID: "de6a1e35-ef8c-4d6f-a845-9caeba54d5e6"). InnerVolumeSpecName "kube-api-access-4gxh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:42:22.173492 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.173472    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56df1516-45a0-4aab-9dbd-b20586a3cea5-kube-api-access-2mjrb" (OuterVolumeSpecName: "kube-api-access-2mjrb") pod "56df1516-45a0-4aab-9dbd-b20586a3cea5" (UID: "56df1516-45a0-4aab-9dbd-b20586a3cea5"). InnerVolumeSpecName "kube-api-access-2mjrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:42:22.272121 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.272075    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mjrb\" (UniqueName: \"kubernetes.io/projected/56df1516-45a0-4aab-9dbd-b20586a3cea5-kube-api-access-2mjrb\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:42:22.272121 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.272116    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gxh2\" (UniqueName: \"kubernetes.io/projected/de6a1e35-ef8c-4d6f-a845-9caeba54d5e6-kube-api-access-4gxh2\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:42:22.272121 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.272129    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56df1516-45a0-4aab-9dbd-b20586a3cea5-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:42:22.272567 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.272143    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6a1e35-ef8c-4d6f-a845-9caeba54d5e6-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:42:22.374462 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.374427    2584 generic.go:356] "Generic (PLEG): container finished" podID="56df1516-45a0-4aab-9dbd-b20586a3cea5" containerID="04c63b5dc50b55c7826ea09a9a18f2826ffd7c472189dda81c1de69f5af9cae0" exitCode=0
Sep 15 22:42:22.374629 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.374487    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"
Sep 15 22:42:22.374629 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.374513    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh" event={"ID":"56df1516-45a0-4aab-9dbd-b20586a3cea5","Type":"ContainerDied","Data":"04c63b5dc50b55c7826ea09a9a18f2826ffd7c472189dda81c1de69f5af9cae0"}
Sep 15 22:42:22.374629 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.374564    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh" event={"ID":"56df1516-45a0-4aab-9dbd-b20586a3cea5","Type":"ContainerDied","Data":"a8350713f3b029144f6b3d11e1b0a6c68a938cc27a274e8fed4c5cc749d2b3f4"}
Sep 15 22:42:22.374629 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.374588    2584 scope.go:117] "RemoveContainer" containerID="04c63b5dc50b55c7826ea09a9a18f2826ffd7c472189dda81c1de69f5af9cae0"
Sep 15 22:42:22.376194 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.376137    2584 generic.go:356] "Generic (PLEG): container finished" podID="de6a1e35-ef8c-4d6f-a845-9caeba54d5e6" containerID="e16cbfd8aa44f2a0bffa298286c6e47e01e6da00a247fc093f57cb8e45b28912" exitCode=0
Sep 15 22:42:22.376345 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.376195    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"
Sep 15 22:42:22.376345 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.376232    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm" event={"ID":"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6","Type":"ContainerDied","Data":"e16cbfd8aa44f2a0bffa298286c6e47e01e6da00a247fc093f57cb8e45b28912"}
Sep 15 22:42:22.376345 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.376297    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm" event={"ID":"de6a1e35-ef8c-4d6f-a845-9caeba54d5e6","Type":"ContainerDied","Data":"b780be7c45562beef7f93abf1f1783b0ddd1bcd8bc9ffebbeeaf34df2aef3e3c"}
Sep 15 22:42:22.384422 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.384395    2584 scope.go:117] "RemoveContainer" containerID="04c63b5dc50b55c7826ea09a9a18f2826ffd7c472189dda81c1de69f5af9cae0"
Sep 15 22:42:22.384728 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:42:22.384711    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c63b5dc50b55c7826ea09a9a18f2826ffd7c472189dda81c1de69f5af9cae0\": container with ID starting with 04c63b5dc50b55c7826ea09a9a18f2826ffd7c472189dda81c1de69f5af9cae0 not found: ID does not exist" containerID="04c63b5dc50b55c7826ea09a9a18f2826ffd7c472189dda81c1de69f5af9cae0"
Sep 15 22:42:22.384773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.384737    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c63b5dc50b55c7826ea09a9a18f2826ffd7c472189dda81c1de69f5af9cae0"} err="failed to get container status \"04c63b5dc50b55c7826ea09a9a18f2826ffd7c472189dda81c1de69f5af9cae0\": rpc error: code = NotFound desc = could not find container \"04c63b5dc50b55c7826ea09a9a18f2826ffd7c472189dda81c1de69f5af9cae0\": container with ID starting with 04c63b5dc50b55c7826ea09a9a18f2826ffd7c472189dda81c1de69f5af9cae0 not found: ID does not exist"
Sep 15 22:42:22.384773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.384753    2584 scope.go:117] "RemoveContainer" containerID="e16cbfd8aa44f2a0bffa298286c6e47e01e6da00a247fc093f57cb8e45b28912"
Sep 15 22:42:22.393456 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.393437    2584 scope.go:117] "RemoveContainer" containerID="e16cbfd8aa44f2a0bffa298286c6e47e01e6da00a247fc093f57cb8e45b28912"
Sep 15 22:42:22.393695 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:42:22.393677    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16cbfd8aa44f2a0bffa298286c6e47e01e6da00a247fc093f57cb8e45b28912\": container with ID starting with e16cbfd8aa44f2a0bffa298286c6e47e01e6da00a247fc093f57cb8e45b28912 not found: ID does not exist" containerID="e16cbfd8aa44f2a0bffa298286c6e47e01e6da00a247fc093f57cb8e45b28912"
Sep 15 22:42:22.393745 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.393701    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16cbfd8aa44f2a0bffa298286c6e47e01e6da00a247fc093f57cb8e45b28912"} err="failed to get container status \"e16cbfd8aa44f2a0bffa298286c6e47e01e6da00a247fc093f57cb8e45b28912\": rpc error: code = NotFound desc = could not find container \"e16cbfd8aa44f2a0bffa298286c6e47e01e6da00a247fc093f57cb8e45b28912\": container with ID starting with e16cbfd8aa44f2a0bffa298286c6e47e01e6da00a247fc093f57cb8e45b28912 not found: ID does not exist"
Sep 15 22:42:22.397546 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.397525    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"]
Sep 15 22:42:22.401620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.401602    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-93550aef-75d7f86f86-jm9zh"]
Sep 15 22:42:22.414387 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.414366    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"]
Sep 15 22:42:22.418118 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.418095    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-93550aef-5b49db7f6c-qdlpm"]
Sep 15 22:42:22.949169 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:22.949134    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:23.121922 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.121830    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:23.527621 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.527542    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:23.795341 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.795309    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"]
Sep 15 22:42:23.795917 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.795892    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de6a1e35-ef8c-4d6f-a845-9caeba54d5e6" containerName="payload-processing"
Sep 15 22:42:23.795917 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.795913    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="de6a1e35-ef8c-4d6f-a845-9caeba54d5e6" containerName="payload-processing"
Sep 15 22:42:23.796125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.795925    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56df1516-45a0-4aab-9dbd-b20586a3cea5" containerName="payload-pre-processing"
Sep 15 22:42:23.796125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.795935    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="56df1516-45a0-4aab-9dbd-b20586a3cea5" containerName="payload-pre-processing"
Sep 15 22:42:23.796125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.795955    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d707bc00-e569-4930-b1d0-b797d72fb41e" containerName="revoke-keys"
Sep 15 22:42:23.796125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.795961    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d707bc00-e569-4930-b1d0-b797d72fb41e" containerName="revoke-keys"
Sep 15 22:42:23.796125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.796046    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="56df1516-45a0-4aab-9dbd-b20586a3cea5" containerName="payload-pre-processing"
Sep 15 22:42:23.796125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.796060    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d707bc00-e569-4930-b1d0-b797d72fb41e" containerName="revoke-keys"
Sep 15 22:42:23.796125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.796073    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="de6a1e35-ef8c-4d6f-a845-9caeba54d5e6" containerName="payload-processing"
Sep 15 22:42:23.800706 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.800685    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.803797 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.803778    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2d1828d0-openshift-default-dockercfg-mzvkr\""
Sep 15 22:42:23.811614 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.811587    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"]
Sep 15 22:42:23.828647 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.828617    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:23.887133 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.887105    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-workload-socket\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.887256 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.887132    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-credential-socket\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.887256 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.887153    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-data\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.887420 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.887297    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-token\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.887420 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.887379    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbfnr\" (UniqueName: \"kubernetes.io/projected/7284ed34-76ef-42ff-b93e-6f5945df2d26-kube-api-access-mbfnr\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.887420 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.887411    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7284ed34-76ef-42ff-b93e-6f5945df2d26-istiod-ca-cert\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.887551 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.887477    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-envoy\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.887551 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.887500    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-workload-certs\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.887551 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.887531    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-podinfo\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.901212 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.901182    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56df1516-45a0-4aab-9dbd-b20586a3cea5" path="/var/lib/kubelet/pods/56df1516-45a0-4aab-9dbd-b20586a3cea5/volumes"
Sep 15 22:42:23.901673 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.901657    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d707bc00-e569-4930-b1d0-b797d72fb41e" path="/var/lib/kubelet/pods/d707bc00-e569-4930-b1d0-b797d72fb41e/volumes"
Sep 15 22:42:23.902118 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.902103    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de6a1e35-ef8c-4d6f-a845-9caeba54d5e6" path="/var/lib/kubelet/pods/de6a1e35-ef8c-4d6f-a845-9caeba54d5e6/volumes"
Sep 15 22:42:23.925699 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.925670    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:23.988301 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.988244    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-workload-socket\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.988301 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.988304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-credential-socket\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.988565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.988323    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-data\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.988565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.988363    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-token\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.988565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.988402    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbfnr\" (UniqueName: \"kubernetes.io/projected/7284ed34-76ef-42ff-b93e-6f5945df2d26-kube-api-access-mbfnr\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.988565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.988430    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7284ed34-76ef-42ff-b93e-6f5945df2d26-istiod-ca-cert\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.988565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.988475    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-envoy\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.988565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.988508    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-workload-certs\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.988565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.988553    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-podinfo\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.988872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.988717    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-credential-socket\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.988910 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.988865    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-workload-socket\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.988910 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.988879    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-workload-certs\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.989006 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.988975    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-data\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.989319 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.989296    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7284ed34-76ef-42ff-b93e-6f5945df2d26-istiod-ca-cert\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.991251 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.991226    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-envoy\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.991251 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.991240    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-podinfo\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:23.998365 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:23.998341    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-token\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:24.001384 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:24.001365    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbfnr\" (UniqueName: \"kubernetes.io/projected/7284ed34-76ef-42ff-b93e-6f5945df2d26-kube-api-access-mbfnr\") pod \"e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") " pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:24.113009 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:24.112926    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:24.399071 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:24.399043    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"]
Sep 15 22:42:24.400656 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:42:24.400629    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7284ed34_76ef_42ff_b93e_6f5945df2d26.slice/crio-ba5712a44bd9769c649eec52eb323023a704998e37266b9b8d11f5b95baa95b6 WatchSource:0}: Error finding container ba5712a44bd9769c649eec52eb323023a704998e37266b9b8d11f5b95baa95b6: Status 404 returned error can't find the container with id ba5712a44bd9769c649eec52eb323023a704998e37266b9b8d11f5b95baa95b6
Sep 15 22:42:24.403179 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:24.403146    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:42:24.403240 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:24.403218    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:42:24.403296 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:24.403248    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:42:24.456804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:24.456772    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:24.582924 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:24.582886    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:25.368433 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:25.368394    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:25.397658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:25.397629    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" event={"ID":"7284ed34-76ef-42ff-b93e-6f5945df2d26","Type":"ContainerStarted","Data":"8285cd117e06005c2062ceb0275a8fde577cb398736cdf14ee1283151e3a8ef5"}
Sep 15 22:42:25.397799 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:25.397685    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" event={"ID":"7284ed34-76ef-42ff-b93e-6f5945df2d26","Type":"ContainerStarted","Data":"ba5712a44bd9769c649eec52eb323023a704998e37266b9b8d11f5b95baa95b6"}
Sep 15 22:42:25.419749 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:25.419700    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" podStartSLOduration=2.419684711 podStartE2EDuration="2.419684711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:42:25.418276328 +0000 UTC m=+918.037091227" watchObservedRunningTime="2026-09-15 22:42:25.419684711 +0000 UTC m=+918.038499618"
Sep 15 22:42:26.113595 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:26.113547    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:26.119045 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:26.119021    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:26.160023 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:26.159992    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:26.403070 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:26.402987    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:26.403956 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:26.403935    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:26.664137 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:26.664045    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:26.763312 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:26.763248    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:26.955425 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:26.955336    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:27.961085 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:27.961052    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:28.679884 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:28.679854    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:28.950005 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:28.949930    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"]
Sep 15 22:42:28.953654 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:28.953631    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:28.957637 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:28.957611    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b08870\"/\"openshift-service-ca.crt\""
Sep 15 22:42:28.957764 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:28.957668    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b08870\"/\"default-dockercfg-q76x8\""
Sep 15 22:42:28.957764 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:28.957682    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b08870\"/\"kube-root-ca.crt\""
Sep 15 22:42:28.957892 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:28.957780    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b08870\"/\"autores-model-b08870-kserve-self-signed-certs\""
Sep 15 22:42:28.963660 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:28.963634    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"]
Sep 15 22:42:28.974507 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:28.974480    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:29.137895 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.137862    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-dshm\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.138072 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.137916    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-home\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.138072 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.137997    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbjb5\" (UniqueName: \"kubernetes.io/projected/c56b92fa-029e-4269-b036-37181df0fb7c-kube-api-access-sbjb5\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.138072 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.138052    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-tmp-dir\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.138245 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.138116    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-model-cache\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.138245 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.138171    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56b92fa-029e-4269-b036-37181df0fb7c-tls-certs\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.239392 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.239282    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbjb5\" (UniqueName: \"kubernetes.io/projected/c56b92fa-029e-4269-b036-37181df0fb7c-kube-api-access-sbjb5\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.239392 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.239380    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-tmp-dir\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.239625 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.239406    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-model-cache\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.239625 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.239469    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56b92fa-029e-4269-b036-37181df0fb7c-tls-certs\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.239625 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.239543    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-dshm\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.239625 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.239583    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-home\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.239894 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.239866    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-tmp-dir\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.240005 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.239899    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-model-cache\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.240005 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.239966    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-home\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.241780 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.241753    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-dshm\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.242033 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.242017    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56b92fa-029e-4269-b036-37181df0fb7c-tls-certs\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.246872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.246851    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbjb5\" (UniqueName: \"kubernetes.io/projected/c56b92fa-029e-4269-b036-37181df0fb7c-kube-api-access-sbjb5\") pod \"autores-model-b08870-kserve-58cd687b75-8bjqm\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") " pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.265051 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.265031    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:29.539335 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.539241    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"]
Sep 15 22:42:29.542771 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:42:29.542740    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc56b92fa_029e_4269_b036_37181df0fb7c.slice/crio-a8040c5a3fe2825aa4ffc10b8aa94073e6d95c487f24bfe487de67fd509370c3 WatchSource:0}: Error finding container a8040c5a3fe2825aa4ffc10b8aa94073e6d95c487f24bfe487de67fd509370c3: Status 404 returned error can't find the container with id a8040c5a3fe2825aa4ffc10b8aa94073e6d95c487f24bfe487de67fd509370c3
Sep 15 22:42:29.555412 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:29.555385    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:30.368305 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:30.368250    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:30.426116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:30.426079    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm" event={"ID":"c56b92fa-029e-4269-b036-37181df0fb7c","Type":"ContainerStarted","Data":"5f4058566b10cb03112fae7bba6b7ff6f0c63ccfaebfa108bec0d7bfd03547b9"}
Sep 15 22:42:30.426287 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:30.426131    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm" event={"ID":"c56b92fa-029e-4269-b036-37181df0fb7c","Type":"ContainerStarted","Data":"a8040c5a3fe2825aa4ffc10b8aa94073e6d95c487f24bfe487de67fd509370c3"}
Sep 15 22:42:30.447417 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:30.447365    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm" podStartSLOduration=2.447349468 podStartE2EDuration="2.447349468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:42:30.445069084 +0000 UTC m=+923.063883992" watchObservedRunningTime="2026-09-15 22:42:30.447349468 +0000 UTC m=+923.066164376"
Sep 15 22:42:30.461705 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:30.461665    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:42:31.080045 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:31.080007    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:31.179040 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:31.178984    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:31.343657 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:31.343571    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:31.570905 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:31.570866    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:31.670390 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:31.670300    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:39.265521 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:39.265480    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:39.265995 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:39.265533    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:39.277844 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:39.277815    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:39.484335 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:39.484309    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:42:42.284222 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:42.284171    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:48.898539 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:48.898043    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:49.363396 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:49.363357    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:42:51.544338 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:51.544305    2584 generic.go:356] "Generic (PLEG): container finished" podID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerID="8285cd117e06005c2062ceb0275a8fde577cb398736cdf14ee1283151e3a8ef5" exitCode=0
Sep 15 22:42:51.544821 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:51.544378    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" event={"ID":"7284ed34-76ef-42ff-b93e-6f5945df2d26","Type":"ContainerDied","Data":"8285cd117e06005c2062ceb0275a8fde577cb398736cdf14ee1283151e3a8ef5"}
Sep 15 22:42:51.544821 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:51.544751    2584 scope.go:117] "RemoveContainer" containerID="8285cd117e06005c2062ceb0275a8fde577cb398736cdf14ee1283151e3a8ef5"
Sep 15 22:42:51.545979 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:51.545944    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:42:51.546082 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:51.546015    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:42:51.546082 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:51.546047    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:42:52.552929 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:52.552895    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" event={"ID":"7284ed34-76ef-42ff-b93e-6f5945df2d26","Type":"ContainerStarted","Data":"22c816a5a3e64976718c77834bb5e4b51bc77efddc427126c84faca01cea6c53"}
Sep 15 22:42:53.113704 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:53.113666    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:53.114779 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:53.114748    2584 patch_prober.go:28] interesting pod/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 15 22:42:53.114921 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:53.114807    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 15 22:42:53.188912 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:53.188869    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:54.113387 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:54.113353    2584 patch_prober.go:28] interesting pod/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 15 22:42:54.113801 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:54.113419    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 15 22:42:54.647378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:54.647335    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:55.113444 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:55.113409    2584 patch_prober.go:28] interesting pod/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 15 22:42:55.113812 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:55.113466    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 15 22:42:55.470686 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:55.470597    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:56.114441 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:56.114401    2584 patch_prober.go:28] interesting pod/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 15 22:42:56.114824 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:56.114474    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 15 22:42:56.245205 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:56.245167    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:56.403820 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:56.403721    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:56.571190 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:56.571158    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj"]
Sep 15 22:42:56.574196 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:56.574165    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj"
Sep 15 22:42:56.578229 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:56.577194    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:42:56.578229 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:56.577445    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:42:56.582722 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:56.582695    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj"]
Sep 15 22:42:56.727643 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:56.727547    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e3804-a93c-4564-8487-0a20e9d2a612-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj\" (UID: \"0b1e3804-a93c-4564-8487-0a20e9d2a612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj"
Sep 15 22:42:56.828867 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:56.828831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e3804-a93c-4564-8487-0a20e9d2a612-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj\" (UID: \"0b1e3804-a93c-4564-8487-0a20e9d2a612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj"
Sep 15 22:42:56.829501 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:56.829481    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e3804-a93c-4564-8487-0a20e9d2a612-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj\" (UID: \"0b1e3804-a93c-4564-8487-0a20e9d2a612\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj"
Sep 15 22:42:56.912227 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:56.912193    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj"
Sep 15 22:42:57.022892 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:57.022779    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:57.114238 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:57.114202    2584 patch_prober.go:28] interesting pod/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 15 22:42:57.114416 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:57.114256    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 15 22:42:57.143284 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:57.143236    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:57.186491 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:57.186452    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj"]
Sep 15 22:42:57.243305 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:57.243277    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:57.585056 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:57.585028    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj" event={"ID":"0b1e3804-a93c-4564-8487-0a20e9d2a612","Type":"ContainerStarted","Data":"1ff22503db8e1b1a6a3f7310114af49f9a3e3f75cbae5a651a81f883cc29a87a"}
Sep 15 22:42:57.900752 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:57.900709    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:58.117769 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:58.117740    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:58.118656 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:58.118635    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:42:58.594018 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:58.593977    2584 generic.go:356] "Generic (PLEG): container finished" podID="0b1e3804-a93c-4564-8487-0a20e9d2a612" containerID="801bb4343c78a4962c4cdc784a3ae0b95c45a026cfc83628ee63d27d9e3fb01d" exitCode=0
Sep 15 22:42:58.594459 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:58.594065    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj" event={"ID":"0b1e3804-a93c-4564-8487-0a20e9d2a612","Type":"ContainerDied","Data":"801bb4343c78a4962c4cdc784a3ae0b95c45a026cfc83628ee63d27d9e3fb01d"}
Sep 15 22:42:58.782235 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:58.782198    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:59.633667 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:59.633623    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:42:59.755620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:59.755595    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj"
Sep 15 22:42:59.858282 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:59.858202    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e3804-a93c-4564-8487-0a20e9d2a612-maas-api-service-ca\") pod \"0b1e3804-a93c-4564-8487-0a20e9d2a612\" (UID: \"0b1e3804-a93c-4564-8487-0a20e9d2a612\") "
Sep 15 22:42:59.858542 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:59.858521    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1e3804-a93c-4564-8487-0a20e9d2a612-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b1e3804-a93c-4564-8487-0a20e9d2a612" (UID: "0b1e3804-a93c-4564-8487-0a20e9d2a612"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:42:59.959492 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:42:59.959466    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e3804-a93c-4564-8487-0a20e9d2a612-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:00.454657 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:00.454617    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:00.605696 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:00.605662    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj"
Sep 15 22:43:00.605865 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:00.605695    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj" event={"ID":"0b1e3804-a93c-4564-8487-0a20e9d2a612","Type":"ContainerDied","Data":"1ff22503db8e1b1a6a3f7310114af49f9a3e3f75cbae5a651a81f883cc29a87a"}
Sep 15 22:43:00.605865 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:00.605743    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ff22503db8e1b1a6a3f7310114af49f9a3e3f75cbae5a651a81f883cc29a87a"
Sep 15 22:43:01.376073 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:01.376031    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:02.070363 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.070327    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:02.321058 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.320978    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph"]
Sep 15 22:43:02.321443 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.321430    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1e3804-a93c-4564-8487-0a20e9d2a612" containerName="revoke-keys"
Sep 15 22:43:02.321497 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.321445    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1e3804-a93c-4564-8487-0a20e9d2a612" containerName="revoke-keys"
Sep 15 22:43:02.324114 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.321947    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1e3804-a93c-4564-8487-0a20e9d2a612" containerName="revoke-keys"
Sep 15 22:43:02.330396 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.330368    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph"
Sep 15 22:43:02.332777 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.332747    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph"]
Sep 15 22:43:02.333934 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.333903    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:43:02.334417 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.333903    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:43:02.486536 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.486503    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f93a436-f9b4-473a-97ac-b9db8fea2c7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b08870-9tqph\" (UID: \"1f93a436-f9b4-473a-97ac-b9db8fea2c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph"
Sep 15 22:43:02.588050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.587954    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f93a436-f9b4-473a-97ac-b9db8fea2c7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b08870-9tqph\" (UID: \"1f93a436-f9b4-473a-97ac-b9db8fea2c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph"
Sep 15 22:43:02.588632 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.588610    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f93a436-f9b4-473a-97ac-b9db8fea2c7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b08870-9tqph\" (UID: \"1f93a436-f9b4-473a-97ac-b9db8fea2c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph"
Sep 15 22:43:02.642975 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.642946    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph"
Sep 15 22:43:02.917366 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.917340    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph"]
Sep 15 22:43:02.919693 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:43:02.919657    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f93a436_f9b4_473a_97ac_b9db8fea2c7d.slice/crio-1981082eece25ac0d25855e3b8b774fd02cb61c423cf3180d3b356c6f0c3bfab WatchSource:0}: Error finding container 1981082eece25ac0d25855e3b8b774fd02cb61c423cf3180d3b356c6f0c3bfab: Status 404 returned error can't find the container with id 1981082eece25ac0d25855e3b8b774fd02cb61c423cf3180d3b356c6f0c3bfab
Sep 15 22:43:02.949836 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:02.949812    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:03.626019 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:03.625987    2584 generic.go:356] "Generic (PLEG): container finished" podID="1f93a436-f9b4-473a-97ac-b9db8fea2c7d" containerID="e3090123db9a5b281034d42d29709beac09052a7fd003f6d0b6f16e276b03fa2" exitCode=0
Sep 15 22:43:03.626391 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:03.626038    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph" event={"ID":"1f93a436-f9b4-473a-97ac-b9db8fea2c7d","Type":"ContainerDied","Data":"e3090123db9a5b281034d42d29709beac09052a7fd003f6d0b6f16e276b03fa2"}
Sep 15 22:43:03.626391 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:03.626103    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph" event={"ID":"1f93a436-f9b4-473a-97ac-b9db8fea2c7d","Type":"ContainerStarted","Data":"1981082eece25ac0d25855e3b8b774fd02cb61c423cf3180d3b356c6f0c3bfab"}
Sep 15 22:43:03.863424 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:03.863332    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:04.537252 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:04.537213    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:04.784460 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:04.784436    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph"
Sep 15 22:43:04.910758 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:04.910726    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f93a436-f9b4-473a-97ac-b9db8fea2c7d-maas-api-service-ca\") pod \"1f93a436-f9b4-473a-97ac-b9db8fea2c7d\" (UID: \"1f93a436-f9b4-473a-97ac-b9db8fea2c7d\") "
Sep 15 22:43:04.911086 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:04.911063    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f93a436-f9b4-473a-97ac-b9db8fea2c7d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f93a436-f9b4-473a-97ac-b9db8fea2c7d" (UID: "1f93a436-f9b4-473a-97ac-b9db8fea2c7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:43:05.012384 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:05.012348    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f93a436-f9b4-473a-97ac-b9db8fea2c7d-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:05.220900 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:05.220801    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:05.639145 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:05.639108    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph" event={"ID":"1f93a436-f9b4-473a-97ac-b9db8fea2c7d","Type":"ContainerDied","Data":"1981082eece25ac0d25855e3b8b774fd02cb61c423cf3180d3b356c6f0c3bfab"}
Sep 15 22:43:05.639347 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:05.639162    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1981082eece25ac0d25855e3b8b774fd02cb61c423cf3180d3b356c6f0c3bfab"
Sep 15 22:43:05.639347 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:05.639127    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph"
Sep 15 22:43:06.065645 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:06.065606    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:06.745465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:06.745424    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:07.515889 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:07.515848    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:08.178124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:08.178080    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:08.278040 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:08.278001    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:08.367736 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:08.367705    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj"]
Sep 15 22:43:08.369782 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:08.369744    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:08.370190 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:08.370170    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2d1828d0-wt7kj"]
Sep 15 22:43:08.424314 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:08.424282    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:08.504582 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:08.504500    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"]
Sep 15 22:43:08.505000 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:08.504969    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerName="istio-proxy" containerID="cri-o://22c816a5a3e64976718c77834bb5e4b51bc77efddc427126c84faca01cea6c53" gracePeriod=30
Sep 15 22:43:08.531124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:08.531083    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:09.193470 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:09.193432    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:09.286044 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:09.285990    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:09.388860 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:09.388826    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:09.486405 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:09.486312    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:09.898770 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:09.898736    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b1e3804-a93c-4564-8487-0a20e9d2a612" path="/var/lib/kubelet/pods/0b1e3804-a93c-4564-8487-0a20e9d2a612/volumes"
Sep 15 22:43:10.175249 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:10.175165    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:10.706023 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:10.705985    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:11.052610 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:11.052569    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:11.730156 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:11.730100    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:12.451217 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:12.451176    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:13.121331 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.121296    2584 patch_prober.go:28] interesting pod/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 22:43:13.121643 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.121594    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 22:43:13.189098 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.189063    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:43:13.196714 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.196674    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:13.226417 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.226383    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"]
Sep 15 22:43:13.226666 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.226640    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj" podUID="8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2" containerName="payload-processing" containerID="cri-o://ce50807c030c7f37dc5fb1f4fd87ea488ed00414b8b278504e7f3284997646b3" gracePeriod=30
Sep 15 22:43:13.265224 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.265192    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:13.364962 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.364926    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:13.385711 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.385636    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph"]
Sep 15 22:43:13.388125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.388095    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b08870-9tqph"]
Sep 15 22:43:13.421219 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.421170    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:13.495355 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.495329    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"
Sep 15 22:43:13.590399 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.590364    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czc9m\" (UniqueName: \"kubernetes.io/projected/8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2-kube-api-access-czc9m\") pod \"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2\" (UID: \"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2\") "
Sep 15 22:43:13.590600 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.590500    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2-plugins-config-volume\") pod \"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2\" (UID: \"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2\") "
Sep 15 22:43:13.590871 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.590822    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2" (UID: "8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:43:13.592649 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.592608    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2-kube-api-access-czc9m" (OuterVolumeSpecName: "kube-api-access-czc9m") pod "8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2" (UID: "8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2"). InnerVolumeSpecName "kube-api-access-czc9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:43:13.688916 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.688879    2584 generic.go:356] "Generic (PLEG): container finished" podID="8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2" containerID="ce50807c030c7f37dc5fb1f4fd87ea488ed00414b8b278504e7f3284997646b3" exitCode=0
Sep 15 22:43:13.689075 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.689062    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj" event={"ID":"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2","Type":"ContainerDied","Data":"ce50807c030c7f37dc5fb1f4fd87ea488ed00414b8b278504e7f3284997646b3"}
Sep 15 22:43:13.689148 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.689113    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj" event={"ID":"8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2","Type":"ContainerDied","Data":"fff4da9379c8f95332116e8725744934eef8f44601782b8881092ec5104a7840"}
Sep 15 22:43:13.689148 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.689141    2584 scope.go:117] "RemoveContainer" containerID="ce50807c030c7f37dc5fb1f4fd87ea488ed00414b8b278504e7f3284997646b3"
Sep 15 22:43:13.689340 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.689322    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"
Sep 15 22:43:13.692087 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.692031    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czc9m\" (UniqueName: \"kubernetes.io/projected/8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2-kube-api-access-czc9m\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:13.692087 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.692060    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:13.696177 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.696149    2584 generic.go:356] "Generic (PLEG): container finished" podID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerID="22c816a5a3e64976718c77834bb5e4b51bc77efddc427126c84faca01cea6c53" exitCode=0
Sep 15 22:43:13.696303 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.696192    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" event={"ID":"7284ed34-76ef-42ff-b93e-6f5945df2d26","Type":"ContainerDied","Data":"22c816a5a3e64976718c77834bb5e4b51bc77efddc427126c84faca01cea6c53"}
Sep 15 22:43:13.707304 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.707261    2584 scope.go:117] "RemoveContainer" containerID="ce50807c030c7f37dc5fb1f4fd87ea488ed00414b8b278504e7f3284997646b3"
Sep 15 22:43:13.707684 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:43:13.707657    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce50807c030c7f37dc5fb1f4fd87ea488ed00414b8b278504e7f3284997646b3\": container with ID starting with ce50807c030c7f37dc5fb1f4fd87ea488ed00414b8b278504e7f3284997646b3 not found: ID does not exist" containerID="ce50807c030c7f37dc5fb1f4fd87ea488ed00414b8b278504e7f3284997646b3"
Sep 15 22:43:13.707784 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.707696    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce50807c030c7f37dc5fb1f4fd87ea488ed00414b8b278504e7f3284997646b3"} err="failed to get container status \"ce50807c030c7f37dc5fb1f4fd87ea488ed00414b8b278504e7f3284997646b3\": rpc error: code = NotFound desc = could not find container \"ce50807c030c7f37dc5fb1f4fd87ea488ed00414b8b278504e7f3284997646b3\": container with ID starting with ce50807c030c7f37dc5fb1f4fd87ea488ed00414b8b278504e7f3284997646b3 not found: ID does not exist"
Sep 15 22:43:13.707784 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.707723    2584 scope.go:117] "RemoveContainer" containerID="8285cd117e06005c2062ceb0275a8fde577cb398736cdf14ee1283151e3a8ef5"
Sep 15 22:43:13.720945 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.720902    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"]
Sep 15 22:43:13.727043 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.727019    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b08870-5d4d6ddcf5-4bzzj"]
Sep 15 22:43:13.728894 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.728871    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:13.756197 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.756179    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:43:13.792610 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.792582    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-credential-socket\") pod \"7284ed34-76ef-42ff-b93e-6f5945df2d26\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") "
Sep 15 22:43:13.792710 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.792611    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbfnr\" (UniqueName: \"kubernetes.io/projected/7284ed34-76ef-42ff-b93e-6f5945df2d26-kube-api-access-mbfnr\") pod \"7284ed34-76ef-42ff-b93e-6f5945df2d26\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") "
Sep 15 22:43:13.792710 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.792651    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7284ed34-76ef-42ff-b93e-6f5945df2d26-istiod-ca-cert\") pod \"7284ed34-76ef-42ff-b93e-6f5945df2d26\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") "
Sep 15 22:43:13.792710 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.792701    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-data\") pod \"7284ed34-76ef-42ff-b93e-6f5945df2d26\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") "
Sep 15 22:43:13.792826 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.792735    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-workload-certs\") pod \"7284ed34-76ef-42ff-b93e-6f5945df2d26\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") "
Sep 15 22:43:13.792826 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.792783    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-podinfo\") pod \"7284ed34-76ef-42ff-b93e-6f5945df2d26\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") "
Sep 15 22:43:13.792923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.792841    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-envoy\") pod \"7284ed34-76ef-42ff-b93e-6f5945df2d26\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") "
Sep 15 22:43:13.792923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.792854    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7284ed34-76ef-42ff-b93e-6f5945df2d26" (UID: "7284ed34-76ef-42ff-b93e-6f5945df2d26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:43:13.792923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.792883    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-workload-socket\") pod \"7284ed34-76ef-42ff-b93e-6f5945df2d26\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") "
Sep 15 22:43:13.792923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.792909    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-token\") pod \"7284ed34-76ef-42ff-b93e-6f5945df2d26\" (UID: \"7284ed34-76ef-42ff-b93e-6f5945df2d26\") "
Sep 15 22:43:13.793122 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.793026    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-data" (OuterVolumeSpecName: "istio-data") pod "7284ed34-76ef-42ff-b93e-6f5945df2d26" (UID: "7284ed34-76ef-42ff-b93e-6f5945df2d26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:43:13.793181 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.793103    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7284ed34-76ef-42ff-b93e-6f5945df2d26" (UID: "7284ed34-76ef-42ff-b93e-6f5945df2d26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:43:13.793236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.793177    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7284ed34-76ef-42ff-b93e-6f5945df2d26" (UID: "7284ed34-76ef-42ff-b93e-6f5945df2d26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:43:13.793367 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.793346    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-workload-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:13.793430 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.793370    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-credential-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:13.793430 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.793384    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-data\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:13.793430 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.793396    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-workload-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:13.793586 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.793554    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7284ed34-76ef-42ff-b93e-6f5945df2d26-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7284ed34-76ef-42ff-b93e-6f5945df2d26" (UID: "7284ed34-76ef-42ff-b93e-6f5945df2d26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:43:13.795005 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.794981    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7284ed34-76ef-42ff-b93e-6f5945df2d26" (UID: "7284ed34-76ef-42ff-b93e-6f5945df2d26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 22:43:13.795124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.795101    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7284ed34-76ef-42ff-b93e-6f5945df2d26-kube-api-access-mbfnr" (OuterVolumeSpecName: "kube-api-access-mbfnr") pod "7284ed34-76ef-42ff-b93e-6f5945df2d26" (UID: "7284ed34-76ef-42ff-b93e-6f5945df2d26"). InnerVolumeSpecName "kube-api-access-mbfnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:43:13.795415 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.795398    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-token" (OuterVolumeSpecName: "istio-token") pod "7284ed34-76ef-42ff-b93e-6f5945df2d26" (UID: "7284ed34-76ef-42ff-b93e-6f5945df2d26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:43:13.795465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.795403    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7284ed34-76ef-42ff-b93e-6f5945df2d26" (UID: "7284ed34-76ef-42ff-b93e-6f5945df2d26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:43:13.894483 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.894448    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-token\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:13.894483 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.894478    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbfnr\" (UniqueName: \"kubernetes.io/projected/7284ed34-76ef-42ff-b93e-6f5945df2d26-kube-api-access-mbfnr\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:13.894677 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.894492    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7284ed34-76ef-42ff-b93e-6f5945df2d26-istiod-ca-cert\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:13.894677 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.894507    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-podinfo\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:13.894677 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.894519    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7284ed34-76ef-42ff-b93e-6f5945df2d26-istio-envoy\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:13.898064 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.898022    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:13.899461 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.899441    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f93a436-f9b4-473a-97ac-b9db8fea2c7d" path="/var/lib/kubelet/pods/1f93a436-f9b4-473a-97ac-b9db8fea2c7d/volumes"
Sep 15 22:43:13.899803 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:13.899789    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2" path="/var/lib/kubelet/pods/8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2/volumes"
Sep 15 22:43:14.596091 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:14.596048    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:14.703414 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:14.703382    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"
Sep 15 22:43:14.703597 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:14.703409    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj" event={"ID":"7284ed34-76ef-42ff-b93e-6f5945df2d26","Type":"ContainerDied","Data":"ba5712a44bd9769c649eec52eb323023a704998e37266b9b8d11f5b95baa95b6"}
Sep 15 22:43:14.703597 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:14.703469    2584 scope.go:117] "RemoveContainer" containerID="22c816a5a3e64976718c77834bb5e4b51bc77efddc427126c84faca01cea6c53"
Sep 15 22:43:14.722505 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:14.722483    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"]
Sep 15 22:43:14.729310 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:14.729259    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2d1828d0-openshift-default-79fc9c5f64-7bcbj"]
Sep 15 22:43:15.288465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:15.288427    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:15.901623 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:15.901589    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" path="/var/lib/kubelet/pods/7284ed34-76ef-42ff-b93e-6f5945df2d26/volumes"
Sep 15 22:43:15.997526 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:15.997492    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:16.531124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:16.531082    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:16.642894 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:16.642852    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:16.733302 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:16.733247    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:16.830796 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:16.830766    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:16.931535 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:16.931496    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:17.286751 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:17.286662    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:18.830598 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:18.830561    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"]
Sep 15 22:43:18.921192 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:18.921157    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:27.843018 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:27.842977    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:31.768400 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:31.768316    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:43:33.850207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:33.850144    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm" podUID="c56b92fa-029e-4269-b036-37181df0fb7c" containerName="main" containerID="cri-o://5f4058566b10cb03112fae7bba6b7ff6f0c63ccfaebfa108bec0d7bfd03547b9" gracePeriod=45
Sep 15 22:43:34.106449 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.106391    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:43:34.185520 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.185488    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-tmp-dir\") pod \"c56b92fa-029e-4269-b036-37181df0fb7c\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") "
Sep 15 22:43:34.185670 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.185593    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-model-cache\") pod \"c56b92fa-029e-4269-b036-37181df0fb7c\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") "
Sep 15 22:43:34.185670 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.185641    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56b92fa-029e-4269-b036-37181df0fb7c-tls-certs\") pod \"c56b92fa-029e-4269-b036-37181df0fb7c\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") "
Sep 15 22:43:34.185763 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.185696    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-home\") pod \"c56b92fa-029e-4269-b036-37181df0fb7c\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") "
Sep 15 22:43:34.185763 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.185732    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-dshm\") pod \"c56b92fa-029e-4269-b036-37181df0fb7c\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") "
Sep 15 22:43:34.185763 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.185757    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbjb5\" (UniqueName: \"kubernetes.io/projected/c56b92fa-029e-4269-b036-37181df0fb7c-kube-api-access-sbjb5\") pod \"c56b92fa-029e-4269-b036-37181df0fb7c\" (UID: \"c56b92fa-029e-4269-b036-37181df0fb7c\") "
Sep 15 22:43:34.185920 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.185753    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c56b92fa-029e-4269-b036-37181df0fb7c" (UID: "c56b92fa-029e-4269-b036-37181df0fb7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:43:34.185920 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.185826    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c56b92fa-029e-4269-b036-37181df0fb7c" (UID: "c56b92fa-029e-4269-b036-37181df0fb7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:43:34.186029 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.185969    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-home" (OuterVolumeSpecName: "home") pod "c56b92fa-029e-4269-b036-37181df0fb7c" (UID: "c56b92fa-029e-4269-b036-37181df0fb7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:43:34.186135 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.186117    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:34.186184 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.186140    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:34.186184 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.186153    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:34.187901 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.187870    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56b92fa-029e-4269-b036-37181df0fb7c-kube-api-access-sbjb5" (OuterVolumeSpecName: "kube-api-access-sbjb5") pod "c56b92fa-029e-4269-b036-37181df0fb7c" (UID: "c56b92fa-029e-4269-b036-37181df0fb7c"). InnerVolumeSpecName "kube-api-access-sbjb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:43:34.187961 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.187882    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-dshm" (OuterVolumeSpecName: "dshm") pod "c56b92fa-029e-4269-b036-37181df0fb7c" (UID: "c56b92fa-029e-4269-b036-37181df0fb7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:43:34.187961 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.187933    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56b92fa-029e-4269-b036-37181df0fb7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c56b92fa-029e-4269-b036-37181df0fb7c" (UID: "c56b92fa-029e-4269-b036-37181df0fb7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:43:34.287040 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.287005    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbjb5\" (UniqueName: \"kubernetes.io/projected/c56b92fa-029e-4269-b036-37181df0fb7c-kube-api-access-sbjb5\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:34.287040 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.287033    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56b92fa-029e-4269-b036-37181df0fb7c-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:34.287040 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.287045    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56b92fa-029e-4269-b036-37181df0fb7c-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:43:34.815020 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.814982    2584 generic.go:356] "Generic (PLEG): container finished" podID="c56b92fa-029e-4269-b036-37181df0fb7c" containerID="5f4058566b10cb03112fae7bba6b7ff6f0c63ccfaebfa108bec0d7bfd03547b9" exitCode=0
Sep 15 22:43:34.815191 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.815030    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"
Sep 15 22:43:34.815191 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.815071    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm" event={"ID":"c56b92fa-029e-4269-b036-37181df0fb7c","Type":"ContainerDied","Data":"5f4058566b10cb03112fae7bba6b7ff6f0c63ccfaebfa108bec0d7bfd03547b9"}
Sep 15 22:43:34.815191 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.815128    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm" event={"ID":"c56b92fa-029e-4269-b036-37181df0fb7c","Type":"ContainerDied","Data":"a8040c5a3fe2825aa4ffc10b8aa94073e6d95c487f24bfe487de67fd509370c3"}
Sep 15 22:43:34.815191 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.815151    2584 scope.go:117] "RemoveContainer" containerID="5f4058566b10cb03112fae7bba6b7ff6f0c63ccfaebfa108bec0d7bfd03547b9"
Sep 15 22:43:34.824815 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.824799    2584 scope.go:117] "RemoveContainer" containerID="5f4058566b10cb03112fae7bba6b7ff6f0c63ccfaebfa108bec0d7bfd03547b9"
Sep 15 22:43:34.825052 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:43:34.825037    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4058566b10cb03112fae7bba6b7ff6f0c63ccfaebfa108bec0d7bfd03547b9\": container with ID starting with 5f4058566b10cb03112fae7bba6b7ff6f0c63ccfaebfa108bec0d7bfd03547b9 not found: ID does not exist" containerID="5f4058566b10cb03112fae7bba6b7ff6f0c63ccfaebfa108bec0d7bfd03547b9"
Sep 15 22:43:34.825092 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.825062    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4058566b10cb03112fae7bba6b7ff6f0c63ccfaebfa108bec0d7bfd03547b9"} err="failed to get container status \"5f4058566b10cb03112fae7bba6b7ff6f0c63ccfaebfa108bec0d7bfd03547b9\": rpc error: code = NotFound desc = could not find container \"5f4058566b10cb03112fae7bba6b7ff6f0c63ccfaebfa108bec0d7bfd03547b9\": container with ID starting with 5f4058566b10cb03112fae7bba6b7ff6f0c63ccfaebfa108bec0d7bfd03547b9 not found: ID does not exist"
Sep 15 22:43:34.837485 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.837464    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"]
Sep 15 22:43:34.840954 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:34.840928    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-b08870/autores-model-b08870-kserve-58cd687b75-8bjqm"]
Sep 15 22:43:35.899444 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:35.899410    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c56b92fa-029e-4269-b036-37181df0fb7c" path="/var/lib/kubelet/pods/c56b92fa-029e-4269-b036-37181df0fb7c/volumes"
Sep 15 22:43:37.773910 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:37.773875    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:43.091808 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:43.091763    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:48.735289 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:48.735230    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:48.840008 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:48.839974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:48.934253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:48.934215    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:55.839066 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:55.839026    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:56.096325 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.096217    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:56.503129 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.503033    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:56.781146 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.781065    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"]
Sep 15 22:43:56.781929 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.781902    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f93a436-f9b4-473a-97ac-b9db8fea2c7d" containerName="revoke-keys"
Sep 15 22:43:56.781929 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.781929    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f93a436-f9b4-473a-97ac-b9db8fea2c7d" containerName="revoke-keys"
Sep 15 22:43:56.782125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.781958    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c56b92fa-029e-4269-b036-37181df0fb7c" containerName="main"
Sep 15 22:43:56.782125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.781968    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c56b92fa-029e-4269-b036-37181df0fb7c" containerName="main"
Sep 15 22:43:56.782125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.781981    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerName="istio-proxy"
Sep 15 22:43:56.782125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.781989    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerName="istio-proxy"
Sep 15 22:43:56.782125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.781998    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerName="istio-proxy"
Sep 15 22:43:56.782125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.782007    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerName="istio-proxy"
Sep 15 22:43:56.782125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.782042    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2" containerName="payload-processing"
Sep 15 22:43:56.782125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.782052    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2" containerName="payload-processing"
Sep 15 22:43:56.782607 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.782139    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f93a436-f9b4-473a-97ac-b9db8fea2c7d" containerName="revoke-keys"
Sep 15 22:43:56.782607 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.782151    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerName="istio-proxy"
Sep 15 22:43:56.782607 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.782162    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c56b92fa-029e-4269-b036-37181df0fb7c" containerName="main"
Sep 15 22:43:56.782607 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.782171    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8832304b-b9c5-4b76-9d84-5f4ab3e1ccc2" containerName="payload-processing"
Sep 15 22:43:56.786979 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.786958    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.789735 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.789715    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-4ccd57-openshift-default-dockercfg-w2jvp\""
Sep 15 22:43:56.794998 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.794792    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-podinfo\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.794998 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.794897    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-token\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.794998 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.794941    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-credential-socket\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.795174 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.795024    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9566\" (UniqueName: \"kubernetes.io/projected/35f7985a-eec4-435f-8b3d-786a9198cbce-kube-api-access-q9566\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.795174 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.795059    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-envoy\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.795174 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.795089    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-data\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.795174 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.795146    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-workload-socket\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.795389 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.795190    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f7985a-eec4-435f-8b3d-786a9198cbce-istiod-ca-cert\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.795389 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.795209    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"]
Sep 15 22:43:56.795389 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.795219    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-workload-certs\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.804720 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.804690    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:56.897236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.895957    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-data\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.897236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.896009    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-workload-socket\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.897236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.896049    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f7985a-eec4-435f-8b3d-786a9198cbce-istiod-ca-cert\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.897236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.896079    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-workload-certs\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.897236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.896313    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-podinfo\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.897236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.896435    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-token\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.897236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.896537    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-data\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.897236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.896553    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-workload-socket\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.897236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.896616    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-credential-socket\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.897236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.896674    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9566\" (UniqueName: \"kubernetes.io/projected/35f7985a-eec4-435f-8b3d-786a9198cbce-kube-api-access-q9566\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.897236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.896711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-envoy\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.897236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.896893    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-workload-certs\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.898349 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.897366    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-credential-socket\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.898349 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.897749    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f7985a-eec4-435f-8b3d-786a9198cbce-istiod-ca-cert\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.900105 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.900056    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-envoy\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.900487 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.900467    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-podinfo\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.904791 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.904766    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-token\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.905243 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.905222    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9566\" (UniqueName: \"kubernetes.io/projected/35f7985a-eec4-435f-8b3d-786a9198cbce-kube-api-access-q9566\") pod \"e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") " pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:56.988970 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:56.988927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:57.099244 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:57.099210    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:57.378189 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:57.378163    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"]
Sep 15 22:43:57.379934 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:43:57.379901    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f7985a_eec4_435f_8b3d_786a9198cbce.slice/crio-95f48fcd39a3a49ac24284b638cd800b31ff9a9c07fe53c0b62e3845d231acbd WatchSource:0}: Error finding container 95f48fcd39a3a49ac24284b638cd800b31ff9a9c07fe53c0b62e3845d231acbd: Status 404 returned error can't find the container with id 95f48fcd39a3a49ac24284b638cd800b31ff9a9c07fe53c0b62e3845d231acbd
Sep 15 22:43:57.382251 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:57.382217    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:43:57.382356 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:57.382303    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:43:57.382356 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:57.382332    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:43:57.951049 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:57.951017    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" event={"ID":"35f7985a-eec4-435f-8b3d-786a9198cbce","Type":"ContainerStarted","Data":"96852955e3dbe97ded30e00f2ca6439e3ad0ca20b8ab7f03add74db4369486be"}
Sep 15 22:43:57.951476 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:57.951063    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" event={"ID":"35f7985a-eec4-435f-8b3d-786a9198cbce","Type":"ContainerStarted","Data":"95f48fcd39a3a49ac24284b638cd800b31ff9a9c07fe53c0b62e3845d231acbd"}
Sep 15 22:43:57.972046 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:57.972001    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" podStartSLOduration=1.971985796 podStartE2EDuration="1.971985796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:43:57.971420411 +0000 UTC m=+1010.590235320" watchObservedRunningTime="2026-09-15 22:43:57.971985796 +0000 UTC m=+1010.590800703"
Sep 15 22:43:58.099437 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:58.099402    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:58.105202 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:58.105178    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:58.956318 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:58.956283    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:58.957346 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:58.957320    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:43:59.227859 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:59.227739    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:59.866997 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:59.866957    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:43:59.965054 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:43:59.965018    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:00.074106 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:00.074068    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:00.870989 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:00.870951    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:01.690846 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:01.690807    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:02.667297 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:02.667248    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:03.364544 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:03.364506    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:03.426909 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:03.426867    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:03.572612 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:03.572569    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:03.833969 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:03.833942    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:03.935651 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:03.935617    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:04.760480 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:04.760436    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:10.471520 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:10.471489    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:10.531019 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:10.530982    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:10.611678 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:10.611641    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:10.711075 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:10.711037    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:13.270608 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:13.270575    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:44:18.109721 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:18.109686    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:18.164556 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:18.164514    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:21.161663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.161626    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:21.572451 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.572416    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"]
Sep 15 22:44:21.573292 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.573255    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7284ed34-76ef-42ff-b93e-6f5945df2d26" containerName="istio-proxy"
Sep 15 22:44:21.578207 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.578182    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.581811 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.581783    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-03f258\"/\"openshift-service-ca.crt\""
Sep 15 22:44:21.582024 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.581995    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-03f258\"/\"default-dockercfg-km8rr\""
Sep 15 22:44:21.582138 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.582026    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-03f258\"/\"premium-w4-03f258-kserve-self-signed-certs\""
Sep 15 22:44:21.582239 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.582220    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-03f258\"/\"kube-root-ca.crt\""
Sep 15 22:44:21.583233 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.583210    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"]
Sep 15 22:44:21.592414 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.592380    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:21.725299 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.725247    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-tmp-dir\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.725572 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.725541    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-model-cache\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.725669 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.725603    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-home\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.725669 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.725632    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01901ba0-7ab7-40ce-a716-512106879cfe-tls-certs\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.725748 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.725678    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62pjf\" (UniqueName: \"kubernetes.io/projected/01901ba0-7ab7-40ce-a716-512106879cfe-kube-api-access-62pjf\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.725817 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.725799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-dshm\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.827216 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.827121    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-dshm\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.827216 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.827176    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-tmp-dir\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.827479 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.827302    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-model-cache\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.827479 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.827359    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-home\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.827479 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.827380    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01901ba0-7ab7-40ce-a716-512106879cfe-tls-certs\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.827479 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.827423    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62pjf\" (UniqueName: \"kubernetes.io/projected/01901ba0-7ab7-40ce-a716-512106879cfe-kube-api-access-62pjf\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.828527 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.828184    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-model-cache\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.828527 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.828481    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-tmp-dir\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.829248 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.829224    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-home\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.831552 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.831525    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-dshm\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.832085 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.832057    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01901ba0-7ab7-40ce-a716-512106879cfe-tls-certs\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.835439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.835414    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62pjf\" (UniqueName: \"kubernetes.io/projected/01901ba0-7ab7-40ce-a716-512106879cfe-kube-api-access-62pjf\") pod \"premium-w4-03f258-kserve-bdd55895b-kkh8t\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") " pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.889309 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.889278    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:21.930552 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:21.930391    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:22.166526 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:22.166363    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"]
Sep 15 22:44:22.169992 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:44:22.169963    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01901ba0_7ab7_40ce_a716_512106879cfe.slice/crio-b8cbe8048e8a252677f877ed90e5854f37a7d305398644dbfbc0935dd6cae8c2 WatchSource:0}: Error finding container b8cbe8048e8a252677f877ed90e5854f37a7d305398644dbfbc0935dd6cae8c2: Status 404 returned error can't find the container with id b8cbe8048e8a252677f877ed90e5854f37a7d305398644dbfbc0935dd6cae8c2
Sep 15 22:44:23.089610 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:23.089578    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t" event={"ID":"01901ba0-7ab7-40ce-a716-512106879cfe","Type":"ContainerStarted","Data":"b96b9b9e52d36e39d6c158c3a1923bc78f56dd3a5051f1b3fa2c94cd4554a9e4"}
Sep 15 22:44:23.089773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:23.089621    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t" event={"ID":"01901ba0-7ab7-40ce-a716-512106879cfe","Type":"ContainerStarted","Data":"b8cbe8048e8a252677f877ed90e5854f37a7d305398644dbfbc0935dd6cae8c2"}
Sep 15 22:44:23.108555 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:23.108511    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t" podStartSLOduration=2.108496451 podStartE2EDuration="2.108496451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:44:23.106740489 +0000 UTC m=+1035.725555397" watchObservedRunningTime="2026-09-15 22:44:23.108496451 +0000 UTC m=+1035.727311359"
Sep 15 22:44:24.097223 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:24.097192    2584 generic.go:356] "Generic (PLEG): container finished" podID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerID="96852955e3dbe97ded30e00f2ca6439e3ad0ca20b8ab7f03add74db4369486be" exitCode=0
Sep 15 22:44:24.097654 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:24.097282    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" event={"ID":"35f7985a-eec4-435f-8b3d-786a9198cbce","Type":"ContainerDied","Data":"96852955e3dbe97ded30e00f2ca6439e3ad0ca20b8ab7f03add74db4369486be"}
Sep 15 22:44:24.097789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:24.097770    2584 scope.go:117] "RemoveContainer" containerID="96852955e3dbe97ded30e00f2ca6439e3ad0ca20b8ab7f03add74db4369486be"
Sep 15 22:44:24.098914 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:24.098885    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:44:24.098980 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:24.098952    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:44:24.099028 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:24.098984    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:44:25.106224 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:25.106186    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" event={"ID":"35f7985a-eec4-435f-8b3d-786a9198cbce","Type":"ContainerStarted","Data":"791ac3e1a8913f2da03305a8dcf461151aa20b0fa479b6109fece7202f5a1abc"}
Sep 15 22:44:25.700178 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:25.700135    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:26.099844 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:26.099803    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:44:26.101195 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:26.101169    2584 patch_prober.go:28] interesting pod/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 15 22:44:26.101321 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:26.101218    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 15 22:44:26.469053 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:26.468968    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:27.099751 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:27.099710    2584 patch_prober.go:28] interesting pod/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 15 22:44:27.099944 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:27.099789    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 15 22:44:27.266929 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:27.266897    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:28.000911 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:28.000863    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:28.067407 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:28.067363    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:28.100054 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:28.100013    2584 patch_prober.go:28] interesting pod/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 15 22:44:28.100345 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:28.100313    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 15 22:44:28.154587 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:28.154548    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:28.252003 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:28.251912    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:28.351780 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:28.351744    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:28.792865 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:28.792810    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:44:28.937484 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:28.937449    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:28.956884 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:28.956857    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:44:29.035994 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:29.035960    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:29.099579 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:29.099546    2584 patch_prober.go:28] interesting pod/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 15 22:44:29.099761 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:29.099605    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 15 22:44:29.142327 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:29.142296    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:29.236441 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:29.236407    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:29.960635 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:29.960595    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:30.100513 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:30.100475    2584 patch_prober.go:28] interesting pod/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 15 22:44:30.101092 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:30.101059    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 15 22:44:30.342909 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:30.342856    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:30.591113 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:30.591081    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:31.104009 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:31.103980    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:44:31.105140 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:31.105120    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:44:31.890117 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:31.890074    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:31.890312 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:31.890138    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:31.903579 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:31.903555    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:32.156404 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:32.156322    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:44:35.591205 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:35.591164    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:35.670794 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:35.670750    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:36.370864 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:36.370818    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:36.470860 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:36.470822    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:36.713101 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:36.713007    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:37.019185 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:37.019098    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:42.860561 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:42.860527    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:42.960228 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:42.960197    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:43.089658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:43.089622    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:43.381460 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:43.381419    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:43.444531 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:43.444500    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:45.369107 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:45.369072    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:44:49.100544 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:49.100505    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:49.392701 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:49.392616    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:49.487450 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:49.487395    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:49.983641 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:49.983602    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:50.758346 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:50.758308    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:51.652597 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:51.652564    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:52.338772 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:52.338736    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:53.065889 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:53.065858    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:53.131219 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:53.131180    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:53.471505 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:53.471412    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:53.742194 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:53.742105    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:53.841901 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:53.841866    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:44:53.887594 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:53.887553    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"]
Sep 15 22:44:53.891483 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:53.891466    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"
Sep 15 22:44:53.894495 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:53.894475    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2bbfe736-dockercfg-mz9tv\""
Sep 15 22:44:53.894575 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:53.894483    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2bbfe736\""
Sep 15 22:44:53.902716 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:53.902694    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"]
Sep 15 22:44:53.952539 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:53.952511    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf58230-81ff-4409-a1b5-6820cfecffe0-plugins-config-volume\") pod \"payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9\" (UID: \"3bf58230-81ff-4409-a1b5-6820cfecffe0\") " pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"
Sep 15 22:44:53.952663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:53.952602    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tzmd\" (UniqueName: \"kubernetes.io/projected/3bf58230-81ff-4409-a1b5-6820cfecffe0-kube-api-access-6tzmd\") pod \"payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9\" (UID: \"3bf58230-81ff-4409-a1b5-6820cfecffe0\") " pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"
Sep 15 22:44:54.053082 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:54.053050    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tzmd\" (UniqueName: \"kubernetes.io/projected/3bf58230-81ff-4409-a1b5-6820cfecffe0-kube-api-access-6tzmd\") pod \"payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9\" (UID: \"3bf58230-81ff-4409-a1b5-6820cfecffe0\") " pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"
Sep 15 22:44:54.053295 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:54.053149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf58230-81ff-4409-a1b5-6820cfecffe0-plugins-config-volume\") pod \"payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9\" (UID: \"3bf58230-81ff-4409-a1b5-6820cfecffe0\") " pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"
Sep 15 22:44:54.053923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:54.053903    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf58230-81ff-4409-a1b5-6820cfecffe0-plugins-config-volume\") pod \"payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9\" (UID: \"3bf58230-81ff-4409-a1b5-6820cfecffe0\") " pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"
Sep 15 22:44:54.061312 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:54.061291    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tzmd\" (UniqueName: \"kubernetes.io/projected/3bf58230-81ff-4409-a1b5-6820cfecffe0-kube-api-access-6tzmd\") pod \"payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9\" (UID: \"3bf58230-81ff-4409-a1b5-6820cfecffe0\") " pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"
Sep 15 22:44:54.203420 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:54.203389    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"
Sep 15 22:44:54.479644 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:54.479612    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"]
Sep 15 22:44:54.481248 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:44:54.481222    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf58230_81ff_4409_a1b5_6820cfecffe0.slice/crio-0a68a8c701e4c5bafd3d8700bbb7ac0f93226ef9c55d45dc175ea67e9d5a9dd8 WatchSource:0}: Error finding container 0a68a8c701e4c5bafd3d8700bbb7ac0f93226ef9c55d45dc175ea67e9d5a9dd8: Status 404 returned error can't find the container with id 0a68a8c701e4c5bafd3d8700bbb7ac0f93226ef9c55d45dc175ea67e9d5a9dd8
Sep 15 22:44:55.276417 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:55.276383    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9" event={"ID":"3bf58230-81ff-4409-a1b5-6820cfecffe0","Type":"ContainerStarted","Data":"6293354e7fefb616cf9a4969dc725a177cac8dcf91fcbf9d7e16268872d7575d"}
Sep 15 22:44:55.276598 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:55.276435    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9" event={"ID":"3bf58230-81ff-4409-a1b5-6820cfecffe0","Type":"ContainerStarted","Data":"0a68a8c701e4c5bafd3d8700bbb7ac0f93226ef9c55d45dc175ea67e9d5a9dd8"}
Sep 15 22:44:55.276598 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:55.276494    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"
Sep 15 22:44:55.293451 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:44:55.293406    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9" podStartSLOduration=2.293389301 podStartE2EDuration="2.293389301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:44:55.292633435 +0000 UTC m=+1067.911448340" watchObservedRunningTime="2026-09-15 22:44:55.293389301 +0000 UTC m=+1067.912204213"
Sep 15 22:45:00.140410 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.140371    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2bbfe736-29825205-vs8ks"]
Sep 15 22:45:00.145065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.145039    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz"]
Sep 15 22:45:00.145306 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.145170    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2bbfe736-29825205-vs8ks"
Sep 15 22:45:00.148358 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.148333    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:45:00.148856 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.148838    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw"]
Sep 15 22:45:00.148959 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.148942    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz"
Sep 15 22:45:00.152153 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.152129    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs"]
Sep 15 22:45:00.152359 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.152217    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw"
Sep 15 22:45:00.157571 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.157548    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2bbfe736-29825205-vs8ks"]
Sep 15 22:45:00.157669 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.157637    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs"
Sep 15 22:45:00.161855 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.161477    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz"]
Sep 15 22:45:00.163741 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.163721    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2bbfe736-29825205-vs8ks"
Sep 15 22:45:00.165856 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.165777    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw"]
Sep 15 22:45:00.166597 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.166580    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss"]
Sep 15 22:45:00.170899 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.170876    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss"
Sep 15 22:45:00.175426 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.175398    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz"
Sep 15 22:45:00.176808 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.176788    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs"]
Sep 15 22:45:00.180502 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.180479    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss"]
Sep 15 22:45:00.224290 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.224226    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw"
Sep 15 22:45:00.237828 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.237138    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs"
Sep 15 22:45:00.281943 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.281619    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss"
Sep 15 22:45:00.456919 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.456893    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2bbfe736-29825205-vs8ks"]
Sep 15 22:45:00.458517 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:45:00.458488    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380a724f_4d41_40be_a4e2_167b9ab1a154.slice/crio-fc92f5920e2af4a8acad52384838a4b729578414e792a83f7e51c136f24d25f8 WatchSource:0}: Error finding container fc92f5920e2af4a8acad52384838a4b729578414e792a83f7e51c136f24d25f8: Status 404 returned error can't find the container with id fc92f5920e2af4a8acad52384838a4b729578414e792a83f7e51c136f24d25f8
Sep 15 22:45:00.465192 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.465168    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz"]
Sep 15 22:45:00.466981 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:45:00.466962    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba140266_cadf_4064_88ad_b062eaf06a6c.slice/crio-c27b0e5cf2e612f2dbaa31782778b05ece8fc4f8ae3ee6e900a5029bb0fe9215 WatchSource:0}: Error finding container c27b0e5cf2e612f2dbaa31782778b05ece8fc4f8ae3ee6e900a5029bb0fe9215: Status 404 returned error can't find the container with id c27b0e5cf2e612f2dbaa31782778b05ece8fc4f8ae3ee6e900a5029bb0fe9215
Sep 15 22:45:00.550302 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.550276    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs"]
Sep 15 22:45:00.551715 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:45:00.551687    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce505db_5217_4724_9496_5c34d460b9cc.slice/crio-84e8a4fb4f01731c8a972dfea89dc00db6d17d5e0e902219295b01ae0a0152da WatchSource:0}: Error finding container 84e8a4fb4f01731c8a972dfea89dc00db6d17d5e0e902219295b01ae0a0152da: Status 404 returned error can't find the container with id 84e8a4fb4f01731c8a972dfea89dc00db6d17d5e0e902219295b01ae0a0152da
Sep 15 22:45:00.552815 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:45:00.552788    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2b94b7_b022_40a5_a771_9b8ae0bce0fd.slice/crio-88afba79b3b7ebe5ad2f00f695bce1f794cae03465903c6830d8debfa3f64bb3 WatchSource:0}: Error finding container 88afba79b3b7ebe5ad2f00f695bce1f794cae03465903c6830d8debfa3f64bb3: Status 404 returned error can't find the container with id 88afba79b3b7ebe5ad2f00f695bce1f794cae03465903c6830d8debfa3f64bb3
Sep 15 22:45:00.553019 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.552993    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw"]
Sep 15 22:45:00.584792 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:00.584769    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss"]
Sep 15 22:45:00.592436 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:45:00.592411    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddf4114_aada_4e11_afe9_389c67dfa6e1.slice/crio-bc2f3459c0c1d3f56422387117de7f26c00a533a1e667dcb24093d7a13f195c2 WatchSource:0}: Error finding container bc2f3459c0c1d3f56422387117de7f26c00a533a1e667dcb24093d7a13f195c2: Status 404 returned error can't find the container with id bc2f3459c0c1d3f56422387117de7f26c00a533a1e667dcb24093d7a13f195c2
Sep 15 22:45:01.326239 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.326196    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs" event={"ID":"bb2b94b7-b022-40a5-a771-9b8ae0bce0fd","Type":"ContainerStarted","Data":"54326748c6408c92cf06f7416618878d2f6eaf54352b06bb01a67934da17e5ab"}
Sep 15 22:45:01.326703 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.326333    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs" event={"ID":"bb2b94b7-b022-40a5-a771-9b8ae0bce0fd","Type":"ContainerStarted","Data":"88afba79b3b7ebe5ad2f00f695bce1f794cae03465903c6830d8debfa3f64bb3"}
Sep 15 22:45:01.327777 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.327751    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz" event={"ID":"ba140266-cadf-4064-88ad-b062eaf06a6c","Type":"ContainerStarted","Data":"9d10469c430757fa28fdd71838e806c97fe7365447aa61582e92ee7c501db077"}
Sep 15 22:45:01.327854 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.327802    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz" event={"ID":"ba140266-cadf-4064-88ad-b062eaf06a6c","Type":"ContainerStarted","Data":"c27b0e5cf2e612f2dbaa31782778b05ece8fc4f8ae3ee6e900a5029bb0fe9215"}
Sep 15 22:45:01.329327 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.329304    2584 generic.go:356] "Generic (PLEG): container finished" podID="380a724f-4d41-40be-a4e2-167b9ab1a154" containerID="0d3914b5166b8cc795d395e83fb928a7191e9db911cb60285729308a33ce3289" exitCode=7
Sep 15 22:45:01.329405 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.329383    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2bbfe736-29825205-vs8ks" event={"ID":"380a724f-4d41-40be-a4e2-167b9ab1a154","Type":"ContainerDied","Data":"0d3914b5166b8cc795d395e83fb928a7191e9db911cb60285729308a33ce3289"}
Sep 15 22:45:01.329461 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.329426    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2bbfe736-29825205-vs8ks" event={"ID":"380a724f-4d41-40be-a4e2-167b9ab1a154","Type":"ContainerStarted","Data":"fc92f5920e2af4a8acad52384838a4b729578414e792a83f7e51c136f24d25f8"}
Sep 15 22:45:01.329515 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.329500    2584 scope.go:117] "RemoveContainer" containerID="0d3914b5166b8cc795d395e83fb928a7191e9db911cb60285729308a33ce3289"
Sep 15 22:45:01.331151 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.331129    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss" event={"ID":"7ddf4114-aada-4e11-afe9-389c67dfa6e1","Type":"ContainerStarted","Data":"81d6ee0cca89ccddd67f487f440ecfba4440ea85cf0f619f2920462cd90e143d"}
Sep 15 22:45:01.331231 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.331169    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss" event={"ID":"7ddf4114-aada-4e11-afe9-389c67dfa6e1","Type":"ContainerStarted","Data":"bc2f3459c0c1d3f56422387117de7f26c00a533a1e667dcb24093d7a13f195c2"}
Sep 15 22:45:01.332789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.332767    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw" event={"ID":"5ce505db-5217-4724-9496-5c34d460b9cc","Type":"ContainerStarted","Data":"e1b4be3cb23a830e64c686cce5686c8f9c45e6d8622e8d7cd7b77b8902213bef"}
Sep 15 22:45:01.332870 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.332811    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw" event={"ID":"5ce505db-5217-4724-9496-5c34d460b9cc","Type":"ContainerStarted","Data":"84e8a4fb4f01731c8a972dfea89dc00db6d17d5e0e902219295b01ae0a0152da"}
Sep 15 22:45:01.342199 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.342155    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs" podStartSLOduration=1.342139448 podStartE2EDuration="1.342139448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:45:01.341431503 +0000 UTC m=+1073.960246411" watchObservedRunningTime="2026-09-15 22:45:01.342139448 +0000 UTC m=+1073.960954375"
Sep 15 22:45:01.357759 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.357708    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz" podStartSLOduration=1.357692292 podStartE2EDuration="1.357692292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:45:01.355489802 +0000 UTC m=+1073.974304711" watchObservedRunningTime="2026-09-15 22:45:01.357692292 +0000 UTC m=+1073.976507258"
Sep 15 22:45:01.386825 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.386776    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss" podStartSLOduration=1.386761457 podStartE2EDuration="1.386761457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:45:01.385917879 +0000 UTC m=+1074.004732788" watchObservedRunningTime="2026-09-15 22:45:01.386761457 +0000 UTC m=+1074.005576365"
Sep 15 22:45:01.400611 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:01.400560    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw" podStartSLOduration=1.400546716 podStartE2EDuration="1.400546716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:45:01.399517034 +0000 UTC m=+1074.018331943" watchObservedRunningTime="2026-09-15 22:45:01.400546716 +0000 UTC m=+1074.019361625"
Sep 15 22:45:02.340560 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:02.340524    2584 generic.go:356] "Generic (PLEG): container finished" podID="7ddf4114-aada-4e11-afe9-389c67dfa6e1" containerID="81d6ee0cca89ccddd67f487f440ecfba4440ea85cf0f619f2920462cd90e143d" exitCode=0
Sep 15 22:45:02.341032 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:02.340613    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss" event={"ID":"7ddf4114-aada-4e11-afe9-389c67dfa6e1","Type":"ContainerDied","Data":"81d6ee0cca89ccddd67f487f440ecfba4440ea85cf0f619f2920462cd90e143d"}
Sep 15 22:45:02.342187 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:02.342166    2584 generic.go:356] "Generic (PLEG): container finished" podID="5ce505db-5217-4724-9496-5c34d460b9cc" containerID="e1b4be3cb23a830e64c686cce5686c8f9c45e6d8622e8d7cd7b77b8902213bef" exitCode=0
Sep 15 22:45:02.342339 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:02.342240    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw" event={"ID":"5ce505db-5217-4724-9496-5c34d460b9cc","Type":"ContainerDied","Data":"e1b4be3cb23a830e64c686cce5686c8f9c45e6d8622e8d7cd7b77b8902213bef"}
Sep 15 22:45:02.343643 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:02.343623    2584 generic.go:356] "Generic (PLEG): container finished" podID="bb2b94b7-b022-40a5-a771-9b8ae0bce0fd" containerID="54326748c6408c92cf06f7416618878d2f6eaf54352b06bb01a67934da17e5ab" exitCode=0
Sep 15 22:45:02.343745 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:02.343689    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs" event={"ID":"bb2b94b7-b022-40a5-a771-9b8ae0bce0fd","Type":"ContainerDied","Data":"54326748c6408c92cf06f7416618878d2f6eaf54352b06bb01a67934da17e5ab"}
Sep 15 22:45:02.345114 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:02.345093    2584 generic.go:356] "Generic (PLEG): container finished" podID="ba140266-cadf-4064-88ad-b062eaf06a6c" containerID="9d10469c430757fa28fdd71838e806c97fe7365447aa61582e92ee7c501db077" exitCode=0
Sep 15 22:45:02.345203 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:02.345118    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz" event={"ID":"ba140266-cadf-4064-88ad-b062eaf06a6c","Type":"ContainerDied","Data":"9d10469c430757fa28fdd71838e806c97fe7365447aa61582e92ee7c501db077"}
Sep 15 22:45:02.346967 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:02.346944    2584 generic.go:356] "Generic (PLEG): container finished" podID="380a724f-4d41-40be-a4e2-167b9ab1a154" containerID="73bb5fa8a52a26772dcf86171b8c7dd60ce8476469e95d1d55c66af2377eeb97" exitCode=0
Sep 15 22:45:02.347065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:02.346991    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2bbfe736-29825205-vs8ks" event={"ID":"380a724f-4d41-40be-a4e2-167b9ab1a154","Type":"ContainerDied","Data":"73bb5fa8a52a26772dcf86171b8c7dd60ce8476469e95d1d55c66af2377eeb97"}
Sep 15 22:45:02.347065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:02.347028    2584 scope.go:117] "RemoveContainer" containerID="0d3914b5166b8cc795d395e83fb928a7191e9db911cb60285729308a33ce3289"
Sep 15 22:45:03.522441 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:03.522413    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2bbfe736-29825205-vs8ks"
Sep 15 22:45:03.584717 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:03.584688    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz"
Sep 15 22:45:03.588708 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:03.588678    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw"
Sep 15 22:45:03.607142 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:03.607110    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs"
Sep 15 22:45:03.613571 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:03.613483    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss"
Sep 15 22:45:03.769634 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:03.769598    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:45:04.367864 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.367837    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss"
Sep 15 22:45:04.368004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.367859    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss" event={"ID":"7ddf4114-aada-4e11-afe9-389c67dfa6e1","Type":"ContainerDied","Data":"bc2f3459c0c1d3f56422387117de7f26c00a533a1e667dcb24093d7a13f195c2"}
Sep 15 22:45:04.368004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.367912    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc2f3459c0c1d3f56422387117de7f26c00a533a1e667dcb24093d7a13f195c2"
Sep 15 22:45:04.369124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.369102    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw"
Sep 15 22:45:04.369243 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.369125    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw" event={"ID":"5ce505db-5217-4724-9496-5c34d460b9cc","Type":"ContainerDied","Data":"84e8a4fb4f01731c8a972dfea89dc00db6d17d5e0e902219295b01ae0a0152da"}
Sep 15 22:45:04.369243 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.369166    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84e8a4fb4f01731c8a972dfea89dc00db6d17d5e0e902219295b01ae0a0152da"
Sep 15 22:45:04.370612 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.370595    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs"
Sep 15 22:45:04.370727 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.370629    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs" event={"ID":"bb2b94b7-b022-40a5-a771-9b8ae0bce0fd","Type":"ContainerDied","Data":"88afba79b3b7ebe5ad2f00f695bce1f794cae03465903c6830d8debfa3f64bb3"}
Sep 15 22:45:04.370727 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.370686    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88afba79b3b7ebe5ad2f00f695bce1f794cae03465903c6830d8debfa3f64bb3"
Sep 15 22:45:04.372212 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.372191    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz"
Sep 15 22:45:04.372334 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.372198    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz" event={"ID":"ba140266-cadf-4064-88ad-b062eaf06a6c","Type":"ContainerDied","Data":"c27b0e5cf2e612f2dbaa31782778b05ece8fc4f8ae3ee6e900a5029bb0fe9215"}
Sep 15 22:45:04.372334 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.372327    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c27b0e5cf2e612f2dbaa31782778b05ece8fc4f8ae3ee6e900a5029bb0fe9215"
Sep 15 22:45:04.373940 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.373918    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2bbfe736-29825205-vs8ks" event={"ID":"380a724f-4d41-40be-a4e2-167b9ab1a154","Type":"ContainerDied","Data":"fc92f5920e2af4a8acad52384838a4b729578414e792a83f7e51c136f24d25f8"}
Sep 15 22:45:04.374044 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.373946    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2bbfe736-29825205-vs8ks"
Sep 15 22:45:04.374044 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:04.373955    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc92f5920e2af4a8acad52384838a4b729578414e792a83f7e51c136f24d25f8"
Sep 15 22:45:06.283935 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:06.283903    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"
Sep 15 22:45:10.165034 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.164997    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"]
Sep 15 22:45:10.165474 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165458    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380a724f-4d41-40be-a4e2-167b9ab1a154" containerName="cleanup"
Sep 15 22:45:10.165474 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165475    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="380a724f-4d41-40be-a4e2-167b9ab1a154" containerName="cleanup"
Sep 15 22:45:10.165565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165486    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2b94b7-b022-40a5-a771-9b8ae0bce0fd" containerName="cleanup"
Sep 15 22:45:10.165565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165492    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2b94b7-b022-40a5-a771-9b8ae0bce0fd" containerName="cleanup"
Sep 15 22:45:10.165565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165501    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ce505db-5217-4724-9496-5c34d460b9cc" containerName="cleanup"
Sep 15 22:45:10.165565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165506    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ce505db-5217-4724-9496-5c34d460b9cc" containerName="cleanup"
Sep 15 22:45:10.165565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165512    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380a724f-4d41-40be-a4e2-167b9ab1a154" containerName="cleanup"
Sep 15 22:45:10.165565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165517    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="380a724f-4d41-40be-a4e2-167b9ab1a154" containerName="cleanup"
Sep 15 22:45:10.165565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165525    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ddf4114-aada-4e11-afe9-389c67dfa6e1" containerName="cleanup"
Sep 15 22:45:10.165565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165531    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ddf4114-aada-4e11-afe9-389c67dfa6e1" containerName="cleanup"
Sep 15 22:45:10.165565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165545    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba140266-cadf-4064-88ad-b062eaf06a6c" containerName="cleanup"
Sep 15 22:45:10.165565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165550    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba140266-cadf-4064-88ad-b062eaf06a6c" containerName="cleanup"
Sep 15 22:45:10.165854 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165625    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2b94b7-b022-40a5-a771-9b8ae0bce0fd" containerName="cleanup"
Sep 15 22:45:10.165854 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165635    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="380a724f-4d41-40be-a4e2-167b9ab1a154" containerName="cleanup"
Sep 15 22:45:10.165854 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165642    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ce505db-5217-4724-9496-5c34d460b9cc" containerName="cleanup"
Sep 15 22:45:10.165854 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165648    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="380a724f-4d41-40be-a4e2-167b9ab1a154" containerName="cleanup"
Sep 15 22:45:10.165854 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165655    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba140266-cadf-4064-88ad-b062eaf06a6c" containerName="cleanup"
Sep 15 22:45:10.165854 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.165662    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ddf4114-aada-4e11-afe9-389c67dfa6e1" containerName="cleanup"
Sep 15 22:45:10.168852 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.168834    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.171673 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.171647    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-765e6d\"/\"kube-root-ca.crt\""
Sep 15 22:45:10.171673 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.171654    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-765e6d\"/\"openshift-service-ca.crt\""
Sep 15 22:45:10.172868 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.172845    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-765e6d\"/\"default-dockercfg-m9wws\""
Sep 15 22:45:10.173019 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.172896    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-765e6d\"/\"test-model-765e6d-kserve-self-signed-certs\""
Sep 15 22:45:10.176868 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.176846    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"]
Sep 15 22:45:10.192777 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.192747    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:10.209695 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.209662    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-dshm\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.209816 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.209711    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86ff26d-da6e-4314-9330-092e2e50e17b-tls-certs\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.209816 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.209744    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-model-cache\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.209816 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.209768    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j25gf\" (UniqueName: \"kubernetes.io/projected/d86ff26d-da6e-4314-9330-092e2e50e17b-kube-api-access-j25gf\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.209816 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.209810    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-home\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.209997 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.209893    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-tmp-dir\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.310794 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.310759    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-dshm\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.310794 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.310803    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86ff26d-da6e-4314-9330-092e2e50e17b-tls-certs\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.311056 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.310896    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-model-cache\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.311056 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.310919    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j25gf\" (UniqueName: \"kubernetes.io/projected/d86ff26d-da6e-4314-9330-092e2e50e17b-kube-api-access-j25gf\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.311056 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.310952    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-home\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.311056 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.311000    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-tmp-dir\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.311385 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.311357    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-model-cache\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.311477 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.311357    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-home\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.311477 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.311429    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-tmp-dir\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.313158 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.313132    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-dshm\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.313714 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.313689    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86ff26d-da6e-4314-9330-092e2e50e17b-tls-certs\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.320162 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.320137    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j25gf\" (UniqueName: \"kubernetes.io/projected/d86ff26d-da6e-4314-9330-092e2e50e17b-kube-api-access-j25gf\") pod \"test-model-765e6d-kserve-b5d6f9b7f-trfst\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") " pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.481312 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.481206    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:10.755323 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:10.755294    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"]
Sep 15 22:45:10.756342 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:45:10.756312    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd86ff26d_da6e_4314_9330_092e2e50e17b.slice/crio-ac58dd447bca56c24e7f9637fe8118641bc46758e42391b7681635989bbcaca9 WatchSource:0}: Error finding container ac58dd447bca56c24e7f9637fe8118641bc46758e42391b7681635989bbcaca9: Status 404 returned error can't find the container with id ac58dd447bca56c24e7f9637fe8118641bc46758e42391b7681635989bbcaca9
Sep 15 22:45:11.417226 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:11.417182    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst" event={"ID":"d86ff26d-da6e-4314-9330-092e2e50e17b","Type":"ContainerStarted","Data":"acabb0a94f92e01ba9a49453eaccdf51927cb02be59b38d403b08bc63e303676"}
Sep 15 22:45:11.417628 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:11.417241    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst" event={"ID":"d86ff26d-da6e-4314-9330-092e2e50e17b","Type":"ContainerStarted","Data":"ac58dd447bca56c24e7f9637fe8118641bc46758e42391b7681635989bbcaca9"}
Sep 15 22:45:11.439160 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:11.439109    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst" podStartSLOduration=1.439094048 podStartE2EDuration="1.439094048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:45:11.436709223 +0000 UTC m=+1084.055524130" watchObservedRunningTime="2026-09-15 22:45:11.439094048 +0000 UTC m=+1084.057908956"
Sep 15 22:45:15.450561 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:15.450522    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:16.150146 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:16.150095    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:16.932724 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:16.932691    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:17.670625 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:17.670583    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:18.569663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:18.569617    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:19.256621 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:19.256584    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:19.413598 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:19.413558    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:19.975817 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:19.975781    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:20.064559 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:20.064520    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:20.410003 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:20.409966    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:20.482149 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:20.482116    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:20.482149 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:20.482158    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:20.495546 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:20.495519    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:21.485726 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:21.485697    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:45:23.074401 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:23.074367    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:45:26.140064 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:26.140022    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:26.241351 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:26.241317    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:26.480883 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:26.480800    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2"]
Sep 15 22:45:26.485681 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:26.485661    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2"
Sep 15 22:45:26.488777 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:26.488751    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:45:26.488913 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:26.488822    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:45:26.492837 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:26.492810    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2"]
Sep 15 22:45:26.564948 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:26.564916    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f14e8c-4b58-4829-a88f-a0ddc373475a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2\" (UID: \"64f14e8c-4b58-4829-a88f-a0ddc373475a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2"
Sep 15 22:45:26.666166 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:26.666135    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f14e8c-4b58-4829-a88f-a0ddc373475a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2\" (UID: \"64f14e8c-4b58-4829-a88f-a0ddc373475a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2"
Sep 15 22:45:26.666844 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:26.666821    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f14e8c-4b58-4829-a88f-a0ddc373475a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2\" (UID: \"64f14e8c-4b58-4829-a88f-a0ddc373475a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2"
Sep 15 22:45:26.800507 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:26.800465    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2"
Sep 15 22:45:27.073158 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:27.073132    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2"]
Sep 15 22:45:27.075382 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:45:27.075353    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f14e8c_4b58_4829_a88f_a0ddc373475a.slice/crio-fb41a17140203a1775c1f1d417d2cb0c32bc8e55718c78e97812203f6df1f4e0 WatchSource:0}: Error finding container fb41a17140203a1775c1f1d417d2cb0c32bc8e55718c78e97812203f6df1f4e0: Status 404 returned error can't find the container with id fb41a17140203a1775c1f1d417d2cb0c32bc8e55718c78e97812203f6df1f4e0
Sep 15 22:45:27.387734 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:27.387705    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:27.512547 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:27.512510    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2" event={"ID":"64f14e8c-4b58-4829-a88f-a0ddc373475a","Type":"ContainerStarted","Data":"7bdf3b990b1fae43c1ebc07301bd3bc307e9f857272e542f88f808e814880f7e"}
Sep 15 22:45:27.512718 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:27.512563    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2" event={"ID":"64f14e8c-4b58-4829-a88f-a0ddc373475a","Type":"ContainerStarted","Data":"fb41a17140203a1775c1f1d417d2cb0c32bc8e55718c78e97812203f6df1f4e0"}
Sep 15 22:45:27.529188 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:27.529142    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2" podStartSLOduration=1.529127058 podStartE2EDuration="1.529127058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:45:27.527538239 +0000 UTC m=+1100.146353157" watchObservedRunningTime="2026-09-15 22:45:27.529127058 +0000 UTC m=+1100.147941966"
Sep 15 22:45:28.097935 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:28.097898    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:28.520513 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:28.520419    2584 generic.go:356] "Generic (PLEG): container finished" podID="64f14e8c-4b58-4829-a88f-a0ddc373475a" containerID="7bdf3b990b1fae43c1ebc07301bd3bc307e9f857272e542f88f808e814880f7e" exitCode=0
Sep 15 22:45:28.520513 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:28.520467    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2" event={"ID":"64f14e8c-4b58-4829-a88f-a0ddc373475a","Type":"ContainerDied","Data":"7bdf3b990b1fae43c1ebc07301bd3bc307e9f857272e542f88f808e814880f7e"}
Sep 15 22:45:28.974882 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:28.974848    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:29.678402 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:29.678377    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2"
Sep 15 22:45:29.679768 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:29.679741    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:29.801632 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:29.801603    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f14e8c-4b58-4829-a88f-a0ddc373475a-maas-api-service-ca\") pod \"64f14e8c-4b58-4829-a88f-a0ddc373475a\" (UID: \"64f14e8c-4b58-4829-a88f-a0ddc373475a\") "
Sep 15 22:45:29.801997 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:29.801970    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f14e8c-4b58-4829-a88f-a0ddc373475a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64f14e8c-4b58-4829-a88f-a0ddc373475a" (UID: "64f14e8c-4b58-4829-a88f-a0ddc373475a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:45:29.902483 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:29.902456    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f14e8c-4b58-4829-a88f-a0ddc373475a-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:45:30.374087 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:30.374034    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:30.533548 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:30.533509    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2" event={"ID":"64f14e8c-4b58-4829-a88f-a0ddc373475a","Type":"ContainerDied","Data":"fb41a17140203a1775c1f1d417d2cb0c32bc8e55718c78e97812203f6df1f4e0"}
Sep 15 22:45:30.533548 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:30.533536    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2"
Sep 15 22:45:30.533743 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:30.533569    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb41a17140203a1775c1f1d417d2cb0c32bc8e55718c78e97812203f6df1f4e0"
Sep 15 22:45:31.074485 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:31.074448    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:31.811406 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:31.811369    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:36.994329 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:36.994284    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:37.000141 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.000112    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2bbfe736-29825205-vs8ks"]
Sep 15 22:45:37.002371 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.002349    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2bbfe736-29825205-vs8ks"]
Sep 15 22:45:37.029773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.029739    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"]
Sep 15 22:45:37.030031 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.029997    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9" podUID="3bf58230-81ff-4409-a1b5-6820cfecffe0" containerName="payload-processing" containerID="cri-o://6293354e7fefb616cf9a4969dc725a177cac8dcf91fcbf9d7e16268872d7575d" gracePeriod=30
Sep 15 22:45:37.074409 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.074306    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:37.169988 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.169952    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:37.209691 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.209625    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2"]
Sep 15 22:45:37.213516 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.213483    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2bbfe736-5ghw2"]
Sep 15 22:45:37.230891 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.230856    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:37.298382 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.298360    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"
Sep 15 22:45:37.374809 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.374765    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tzmd\" (UniqueName: \"kubernetes.io/projected/3bf58230-81ff-4409-a1b5-6820cfecffe0-kube-api-access-6tzmd\") pod \"3bf58230-81ff-4409-a1b5-6820cfecffe0\" (UID: \"3bf58230-81ff-4409-a1b5-6820cfecffe0\") "
Sep 15 22:45:37.376090 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.376042    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf58230-81ff-4409-a1b5-6820cfecffe0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3bf58230-81ff-4409-a1b5-6820cfecffe0" (UID: "3bf58230-81ff-4409-a1b5-6820cfecffe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:45:37.376301 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.376280    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf58230-81ff-4409-a1b5-6820cfecffe0-plugins-config-volume\") pod \"3bf58230-81ff-4409-a1b5-6820cfecffe0\" (UID: \"3bf58230-81ff-4409-a1b5-6820cfecffe0\") "
Sep 15 22:45:37.376864 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.376841    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf58230-81ff-4409-a1b5-6820cfecffe0-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:45:37.378594 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.378558    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf58230-81ff-4409-a1b5-6820cfecffe0-kube-api-access-6tzmd" (OuterVolumeSpecName: "kube-api-access-6tzmd") pod "3bf58230-81ff-4409-a1b5-6820cfecffe0" (UID: "3bf58230-81ff-4409-a1b5-6820cfecffe0"). InnerVolumeSpecName "kube-api-access-6tzmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:45:37.477567 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.477526    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tzmd\" (UniqueName: \"kubernetes.io/projected/3bf58230-81ff-4409-a1b5-6820cfecffe0-kube-api-access-6tzmd\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:45:37.561608 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.561569    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:37.573805 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.573775    2584 generic.go:356] "Generic (PLEG): container finished" podID="3bf58230-81ff-4409-a1b5-6820cfecffe0" containerID="6293354e7fefb616cf9a4969dc725a177cac8dcf91fcbf9d7e16268872d7575d" exitCode=0
Sep 15 22:45:37.573977 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.573822    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"
Sep 15 22:45:37.573977 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.573836    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9" event={"ID":"3bf58230-81ff-4409-a1b5-6820cfecffe0","Type":"ContainerDied","Data":"6293354e7fefb616cf9a4969dc725a177cac8dcf91fcbf9d7e16268872d7575d"}
Sep 15 22:45:37.573977 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.573890    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9" event={"ID":"3bf58230-81ff-4409-a1b5-6820cfecffe0","Type":"ContainerDied","Data":"0a68a8c701e4c5bafd3d8700bbb7ac0f93226ef9c55d45dc175ea67e9d5a9dd8"}
Sep 15 22:45:37.573977 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.573919    2584 scope.go:117] "RemoveContainer" containerID="6293354e7fefb616cf9a4969dc725a177cac8dcf91fcbf9d7e16268872d7575d"
Sep 15 22:45:37.585343 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.585323    2584 scope.go:117] "RemoveContainer" containerID="6293354e7fefb616cf9a4969dc725a177cac8dcf91fcbf9d7e16268872d7575d"
Sep 15 22:45:37.585662 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:45:37.585638    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6293354e7fefb616cf9a4969dc725a177cac8dcf91fcbf9d7e16268872d7575d\": container with ID starting with 6293354e7fefb616cf9a4969dc725a177cac8dcf91fcbf9d7e16268872d7575d not found: ID does not exist" containerID="6293354e7fefb616cf9a4969dc725a177cac8dcf91fcbf9d7e16268872d7575d"
Sep 15 22:45:37.585744 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.585670    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6293354e7fefb616cf9a4969dc725a177cac8dcf91fcbf9d7e16268872d7575d"} err="failed to get container status \"6293354e7fefb616cf9a4969dc725a177cac8dcf91fcbf9d7e16268872d7575d\": rpc error: code = NotFound desc = could not find container \"6293354e7fefb616cf9a4969dc725a177cac8dcf91fcbf9d7e16268872d7575d\": container with ID starting with 6293354e7fefb616cf9a4969dc725a177cac8dcf91fcbf9d7e16268872d7575d not found: ID does not exist"
Sep 15 22:45:37.598136 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.598109    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"]
Sep 15 22:45:37.602169 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.602144    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2bbfe736-6f4f7779cc-6rcc9"]
Sep 15 22:45:37.664599 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.664564    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:37.899440 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.899359    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="380a724f-4d41-40be-a4e2-167b9ab1a154" path="/var/lib/kubelet/pods/380a724f-4d41-40be-a4e2-167b9ab1a154/volumes"
Sep 15 22:45:37.899719 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.899706    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf58230-81ff-4409-a1b5-6820cfecffe0" path="/var/lib/kubelet/pods/3bf58230-81ff-4409-a1b5-6820cfecffe0/volumes"
Sep 15 22:45:37.900023 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:37.900010    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64f14e8c-4b58-4829-a88f-a0ddc373475a" path="/var/lib/kubelet/pods/64f14e8c-4b58-4829-a88f-a0ddc373475a/volumes"
Sep 15 22:45:38.484772 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:38.484731    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:39.379534 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:39.379493    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:40.069710 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:40.069676    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:40.934223 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:40.934192    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:41.765854 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:41.765806    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:42.862759 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:42.862724    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:45:43.457387 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:43.457351    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:43.524884 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:43.524850    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:43.699501 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:43.699460    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:44.049848 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:44.049809    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:44.151478 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:44.151439    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:44.248456 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:44.248424    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:46.964964 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.964927    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"]
Sep 15 22:45:46.965616 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.965595    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f14e8c-4b58-4829-a88f-a0ddc373475a" containerName="revoke-keys"
Sep 15 22:45:46.965664 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.965618    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f14e8c-4b58-4829-a88f-a0ddc373475a" containerName="revoke-keys"
Sep 15 22:45:46.965664 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.965653    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf58230-81ff-4409-a1b5-6820cfecffe0" containerName="payload-processing"
Sep 15 22:45:46.965664 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.965661    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf58230-81ff-4409-a1b5-6820cfecffe0" containerName="payload-processing"
Sep 15 22:45:46.965789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.965777    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f14e8c-4b58-4829-a88f-a0ddc373475a" containerName="revoke-keys"
Sep 15 22:45:46.965828 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.965799    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf58230-81ff-4409-a1b5-6820cfecffe0" containerName="payload-processing"
Sep 15 22:45:46.970642 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.970619    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:46.975079 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.975049    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4ccd57\"/\"openshift-service-ca.crt\""
Sep 15 22:45:46.975349 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.975256    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4ccd57\"/\"explicit-model-4ccd57-kserve-self-signed-certs\""
Sep 15 22:45:46.976082 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.975260    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4ccd57\"/\"kube-root-ca.crt\""
Sep 15 22:45:46.976082 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.975410    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4ccd57\"/\"default-dockercfg-h4gf4\""
Sep 15 22:45:46.979773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.979752    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"]
Sep 15 22:45:46.995888 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:46.995829    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:47.073031 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.072997    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-home\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.073178 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.073041    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9skhv\" (UniqueName: \"kubernetes.io/projected/6786229b-290f-4a44-b51f-bc654aed9a99-kube-api-access-9skhv\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.073178 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.073128    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-dshm\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.073254 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.073181    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-model-cache\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.073254 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.073204    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6786229b-290f-4a44-b51f-bc654aed9a99-tls-certs\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.073341 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.073289    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-tmp-dir\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.174026 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.173994    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-dshm\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.174219 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.174041    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-model-cache\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.174219 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.174057    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6786229b-290f-4a44-b51f-bc654aed9a99-tls-certs\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.174219 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.174110    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-tmp-dir\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.174219 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.174183    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-home\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.174219 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.174206    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9skhv\" (UniqueName: \"kubernetes.io/projected/6786229b-290f-4a44-b51f-bc654aed9a99-kube-api-access-9skhv\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.174583 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.174550    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-model-cache\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.174583 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.174563    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-home\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.174719 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.174638    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-tmp-dir\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.176199 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.176177    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-dshm\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.176619 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.176596    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6786229b-290f-4a44-b51f-bc654aed9a99-tls-certs\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.182766 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.182744    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9skhv\" (UniqueName: \"kubernetes.io/projected/6786229b-290f-4a44-b51f-bc654aed9a99-kube-api-access-9skhv\") pod \"explicit-model-4ccd57-kserve-777895cd4f-52qwn\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") " pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.285411 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.285328    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:47.563281 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.563240    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"]
Sep 15 22:45:47.564064 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:45:47.564035    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6786229b_290f_4a44_b51f_bc654aed9a99.slice/crio-48473fc20f1edaaf60cda614a8ad24efacbd20e30e9fa582dd1921f4920a256d WatchSource:0}: Error finding container 48473fc20f1edaaf60cda614a8ad24efacbd20e30e9fa582dd1921f4920a256d: Status 404 returned error can't find the container with id 48473fc20f1edaaf60cda614a8ad24efacbd20e30e9fa582dd1921f4920a256d
Sep 15 22:45:47.635715 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.635689    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn" event={"ID":"6786229b-290f-4a44-b51f-bc654aed9a99","Type":"ContainerStarted","Data":"0b831f9a8a9e1f23e4702a9956d942e5cda85b42beef59c94dadb7d458bc6103"}
Sep 15 22:45:47.635826 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.635730    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn" event={"ID":"6786229b-290f-4a44-b51f-bc654aed9a99","Type":"ContainerStarted","Data":"48473fc20f1edaaf60cda614a8ad24efacbd20e30e9fa582dd1921f4920a256d"}
Sep 15 22:45:47.652483 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:47.652443    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn" podStartSLOduration=1.6524276530000002 podStartE2EDuration="1.652427653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:45:47.651703221 +0000 UTC m=+1120.270518131" watchObservedRunningTime="2026-09-15 22:45:47.652427653 +0000 UTC m=+1120.271242561"
Sep 15 22:45:48.760001 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:48.759969    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:50.286192 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:50.286144    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:50.386176 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:50.386136    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:50.499124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:50.499086    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:51.268820 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:51.268780    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:52.175337 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:52.175302    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:52.276010 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:52.275976    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:52.870687 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:52.870650    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:56.452250 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:56.452207    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:56.551643 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:56.551607    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:57.285990 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:57.285953    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:57.286174 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:57.286027    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:57.298787 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:57.298766    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:57.336462 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:57.336437    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:57.645899 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:57.645858    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:45:57.704093 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:57.704065    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:45:57.989650 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:45:57.989562    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:00.172378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:00.172344    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:00.733505 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:00.733459    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:00.830989 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:00.830953    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:00.930004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:00.929968    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:01.779714 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:01.779652    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:02.568383 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:02.568346    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:03.383449 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:03.383412    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:04.072517 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:04.072479    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:04.954888 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:04.954855    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:05.081435 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.081398    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:05.138230 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.138202    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:05.443353 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.443315    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:05.514548 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.514508    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:05.591884 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.591843    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"]
Sep 15 22:46:05.596458 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.596440    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"
Sep 15 22:46:05.599978 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.599952    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-f2241f-dockercfg-dsvs6\""
Sep 15 22:46:05.600088 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.599966    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-f2241f\""
Sep 15 22:46:05.604217 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.604194    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"]
Sep 15 22:46:05.652618 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.652591    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdll6\" (UniqueName: \"kubernetes.io/projected/73971904-e99b-4662-a71b-0ee3b26640d5-kube-api-access-sdll6\") pod \"payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54\" (UID: \"73971904-e99b-4662-a71b-0ee3b26640d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"
Sep 15 22:46:05.652743 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.652623    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73971904-e99b-4662-a71b-0ee3b26640d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54\" (UID: \"73971904-e99b-4662-a71b-0ee3b26640d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"
Sep 15 22:46:05.753960 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.753886    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdll6\" (UniqueName: \"kubernetes.io/projected/73971904-e99b-4662-a71b-0ee3b26640d5-kube-api-access-sdll6\") pod \"payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54\" (UID: \"73971904-e99b-4662-a71b-0ee3b26640d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"
Sep 15 22:46:05.753960 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.753925    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73971904-e99b-4662-a71b-0ee3b26640d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54\" (UID: \"73971904-e99b-4662-a71b-0ee3b26640d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"
Sep 15 22:46:05.754726 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.754705    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73971904-e99b-4662-a71b-0ee3b26640d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54\" (UID: \"73971904-e99b-4662-a71b-0ee3b26640d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"
Sep 15 22:46:05.763301 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.763259    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdll6\" (UniqueName: \"kubernetes.io/projected/73971904-e99b-4662-a71b-0ee3b26640d5-kube-api-access-sdll6\") pod \"payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54\" (UID: \"73971904-e99b-4662-a71b-0ee3b26640d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"
Sep 15 22:46:05.908963 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:05.908931    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"
Sep 15 22:46:06.181124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:06.181092    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"]
Sep 15 22:46:06.182438 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:46:06.182409    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73971904_e99b_4662_a71b_0ee3b26640d5.slice/crio-3bb84f9e3a54da901d1005a1d2061fbaf0068c4eb884bf2d8119e5fcc7e386a9 WatchSource:0}: Error finding container 3bb84f9e3a54da901d1005a1d2061fbaf0068c4eb884bf2d8119e5fcc7e386a9: Status 404 returned error can't find the container with id 3bb84f9e3a54da901d1005a1d2061fbaf0068c4eb884bf2d8119e5fcc7e386a9
Sep 15 22:46:06.746661 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:06.746620    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54" event={"ID":"73971904-e99b-4662-a71b-0ee3b26640d5","Type":"ContainerStarted","Data":"99501850696cb9dd7b5210648c1cac6689b3e6366da91523833ca951abe27f30"}
Sep 15 22:46:06.746853 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:06.746673    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54" event={"ID":"73971904-e99b-4662-a71b-0ee3b26640d5","Type":"ContainerStarted","Data":"3bb84f9e3a54da901d1005a1d2061fbaf0068c4eb884bf2d8119e5fcc7e386a9"}
Sep 15 22:46:06.746853 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:06.746728    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"
Sep 15 22:46:06.763107 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:06.763056    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54" podStartSLOduration=1.763039725 podStartE2EDuration="1.763039725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:46:06.762643977 +0000 UTC m=+1139.381458881" watchObservedRunningTime="2026-09-15 22:46:06.763039725 +0000 UTC m=+1139.381854634"
Sep 15 22:46:07.761174 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:07.761144    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:46:11.943586 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:11.943546    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"]
Sep 15 22:46:11.957698 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:11.957670    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:12.070562 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:12.070526    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z"]
Sep 15 22:46:12.075656 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:12.075637    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z"
Sep 15 22:46:12.079254 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:12.079228    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:46:12.079517 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:12.079497    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:46:12.081958 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:12.081932    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z"]
Sep 15 22:46:12.212708 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:12.212621    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8bb31a-060e-445d-a626-ed98ed30f289-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z\" (UID: \"ee8bb31a-060e-445d-a626-ed98ed30f289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z"
Sep 15 22:46:12.313852 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:12.313811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8bb31a-060e-445d-a626-ed98ed30f289-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z\" (UID: \"ee8bb31a-060e-445d-a626-ed98ed30f289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z"
Sep 15 22:46:12.314476 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:12.314455    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8bb31a-060e-445d-a626-ed98ed30f289-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z\" (UID: \"ee8bb31a-060e-445d-a626-ed98ed30f289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z"
Sep 15 22:46:12.331414 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:12.331383    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:12.389393 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:12.389368    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z"
Sep 15 22:46:12.666712 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:12.666689    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z"]
Sep 15 22:46:12.669093 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:46:12.669066    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8bb31a_060e_445d_a626_ed98ed30f289.slice/crio-f101a25c34eb9e1240a60c91f96b27e3d8101481c838f9d3c695ccd9d4e5321c WatchSource:0}: Error finding container f101a25c34eb9e1240a60c91f96b27e3d8101481c838f9d3c695ccd9d4e5321c: Status 404 returned error can't find the container with id f101a25c34eb9e1240a60c91f96b27e3d8101481c838f9d3c695ccd9d4e5321c
Sep 15 22:46:12.790633 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:12.790600    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z" event={"ID":"ee8bb31a-060e-445d-a626-ed98ed30f289","Type":"ContainerStarted","Data":"f101a25c34eb9e1240a60c91f96b27e3d8101481c838f9d3c695ccd9d4e5321c"}
Sep 15 22:46:13.187582 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:13.187548    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:13.798188 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:13.798156    2584 generic.go:356] "Generic (PLEG): container finished" podID="ee8bb31a-060e-445d-a626-ed98ed30f289" containerID="b50cca19dede60109d9276da15e4badeae6e610f5041d193e98a019504598b63" exitCode=0
Sep 15 22:46:13.798343 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:13.798226    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z" event={"ID":"ee8bb31a-060e-445d-a626-ed98ed30f289","Type":"ContainerDied","Data":"b50cca19dede60109d9276da15e4badeae6e610f5041d193e98a019504598b63"}
Sep 15 22:46:13.986890 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:13.986855    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:14.667112 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:14.667067    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:14.945517 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:14.945487    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z"
Sep 15 22:46:15.038178 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:15.038145    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8bb31a-060e-445d-a626-ed98ed30f289-maas-api-service-ca\") pod \"ee8bb31a-060e-445d-a626-ed98ed30f289\" (UID: \"ee8bb31a-060e-445d-a626-ed98ed30f289\") "
Sep 15 22:46:15.038581 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:15.038561    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8bb31a-060e-445d-a626-ed98ed30f289-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ee8bb31a-060e-445d-a626-ed98ed30f289" (UID: "ee8bb31a-060e-445d-a626-ed98ed30f289"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:46:15.140029 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:15.140000    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8bb31a-060e-445d-a626-ed98ed30f289-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:15.454773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:15.454743    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:15.811577 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:15.811542    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z" event={"ID":"ee8bb31a-060e-445d-a626-ed98ed30f289","Type":"ContainerDied","Data":"f101a25c34eb9e1240a60c91f96b27e3d8101481c838f9d3c695ccd9d4e5321c"}
Sep 15 22:46:15.811577 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:15.811568    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z"
Sep 15 22:46:15.812011 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:15.811593    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f101a25c34eb9e1240a60c91f96b27e3d8101481c838f9d3c695ccd9d4e5321c"
Sep 15 22:46:16.160778 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:16.160698    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:16.954391 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:16.954360    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:17.627184 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:17.627147    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:17.754459 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:17.754424    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"
Sep 15 22:46:18.462341 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:18.462304    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:19.258545 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:19.258513    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:19.955341 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:19.955305    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:20.545120 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:20.545081    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:20.802828 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:20.802740    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:20.902594 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:20.902557    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:21.159300 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:21.159155    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:21.258708 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:21.258677    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:21.556131 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:21.556096    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:22.292030 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:22.291997    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs"]
Sep 15 22:46:22.292926 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:22.292889    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:22.294653 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:22.294633    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ccd57-29825205-wkjbs"]
Sep 15 22:46:22.360183 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:22.360148    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:22.459926 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:22.459884    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:22.484121 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:22.484095    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z"]
Sep 15 22:46:22.487021 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:22.486999    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4ccd57-sm96z"]
Sep 15 22:46:22.523342 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:22.523317    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:22.835764 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:22.835736    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:22.937294 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:22.937237    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:23.777473 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:23.777442    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:23.899893 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:23.899859    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2b94b7-b022-40a5-a771-9b8ae0bce0fd" path="/var/lib/kubelet/pods/bb2b94b7-b022-40a5-a771-9b8ae0bce0fd/volumes"
Sep 15 22:46:23.900262 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:23.900247    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee8bb31a-060e-445d-a626-ed98ed30f289" path="/var/lib/kubelet/pods/ee8bb31a-060e-445d-a626-ed98ed30f289/volumes"
Sep 15 22:46:23.984729 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:23.984649    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:24.079316 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:24.079281    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:24.602160 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:24.602130    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:25.540715 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:25.540685    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:26.346532 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:26.346500    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:26.964873 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:26.964823    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn" podUID="6786229b-290f-4a44-b51f-bc654aed9a99" containerName="main" containerID="cri-o://0b831f9a8a9e1f23e4702a9956d942e5cda85b42beef59c94dadb7d458bc6103" gracePeriod=45
Sep 15 22:46:27.177156 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.177118    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:27.231071 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.231018    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:46:27.361225 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.361196    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9skhv\" (UniqueName: \"kubernetes.io/projected/6786229b-290f-4a44-b51f-bc654aed9a99-kube-api-access-9skhv\") pod \"6786229b-290f-4a44-b51f-bc654aed9a99\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") "
Sep 15 22:46:27.361405 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.361283    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-model-cache\") pod \"6786229b-290f-4a44-b51f-bc654aed9a99\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") "
Sep 15 22:46:27.361405 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.361341    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-dshm\") pod \"6786229b-290f-4a44-b51f-bc654aed9a99\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") "
Sep 15 22:46:27.361405 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.361358    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-home\") pod \"6786229b-290f-4a44-b51f-bc654aed9a99\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") "
Sep 15 22:46:27.361570 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.361414    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6786229b-290f-4a44-b51f-bc654aed9a99-tls-certs\") pod \"6786229b-290f-4a44-b51f-bc654aed9a99\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") "
Sep 15 22:46:27.361570 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.361429    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-tmp-dir\") pod \"6786229b-290f-4a44-b51f-bc654aed9a99\" (UID: \"6786229b-290f-4a44-b51f-bc654aed9a99\") "
Sep 15 22:46:27.361680 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.361572    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-model-cache" (OuterVolumeSpecName: "model-cache") pod "6786229b-290f-4a44-b51f-bc654aed9a99" (UID: "6786229b-290f-4a44-b51f-bc654aed9a99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:46:27.361738 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.361668    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-home" (OuterVolumeSpecName: "home") pod "6786229b-290f-4a44-b51f-bc654aed9a99" (UID: "6786229b-290f-4a44-b51f-bc654aed9a99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:46:27.361738 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.361721    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6786229b-290f-4a44-b51f-bc654aed9a99" (UID: "6786229b-290f-4a44-b51f-bc654aed9a99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:46:27.361836 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.361761    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:27.363514 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.363484    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6786229b-290f-4a44-b51f-bc654aed9a99-kube-api-access-9skhv" (OuterVolumeSpecName: "kube-api-access-9skhv") pod "6786229b-290f-4a44-b51f-bc654aed9a99" (UID: "6786229b-290f-4a44-b51f-bc654aed9a99"). InnerVolumeSpecName "kube-api-access-9skhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:46:27.363514 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.363496    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-dshm" (OuterVolumeSpecName: "dshm") pod "6786229b-290f-4a44-b51f-bc654aed9a99" (UID: "6786229b-290f-4a44-b51f-bc654aed9a99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:46:27.363663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.363530    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6786229b-290f-4a44-b51f-bc654aed9a99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6786229b-290f-4a44-b51f-bc654aed9a99" (UID: "6786229b-290f-4a44-b51f-bc654aed9a99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:46:27.462976 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.462941    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:27.462976 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.462972    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:27.463155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.462986    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6786229b-290f-4a44-b51f-bc654aed9a99-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:27.463155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.463001    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6786229b-290f-4a44-b51f-bc654aed9a99-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:27.463155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.463013    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9skhv\" (UniqueName: \"kubernetes.io/projected/6786229b-290f-4a44-b51f-bc654aed9a99-kube-api-access-9skhv\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:27.883924 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.883883    2584 generic.go:356] "Generic (PLEG): container finished" podID="6786229b-290f-4a44-b51f-bc654aed9a99" containerID="0b831f9a8a9e1f23e4702a9956d942e5cda85b42beef59c94dadb7d458bc6103" exitCode=0
Sep 15 22:46:27.884124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.883932    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"
Sep 15 22:46:27.884124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.883971    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn" event={"ID":"6786229b-290f-4a44-b51f-bc654aed9a99","Type":"ContainerDied","Data":"0b831f9a8a9e1f23e4702a9956d942e5cda85b42beef59c94dadb7d458bc6103"}
Sep 15 22:46:27.884124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.884021    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn" event={"ID":"6786229b-290f-4a44-b51f-bc654aed9a99","Type":"ContainerDied","Data":"48473fc20f1edaaf60cda614a8ad24efacbd20e30e9fa582dd1921f4920a256d"}
Sep 15 22:46:27.884124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.884042    2584 scope.go:117] "RemoveContainer" containerID="0b831f9a8a9e1f23e4702a9956d942e5cda85b42beef59c94dadb7d458bc6103"
Sep 15 22:46:27.894112 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.894085    2584 scope.go:117] "RemoveContainer" containerID="0b831f9a8a9e1f23e4702a9956d942e5cda85b42beef59c94dadb7d458bc6103"
Sep 15 22:46:27.894425 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:46:27.894403    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b831f9a8a9e1f23e4702a9956d942e5cda85b42beef59c94dadb7d458bc6103\": container with ID starting with 0b831f9a8a9e1f23e4702a9956d942e5cda85b42beef59c94dadb7d458bc6103 not found: ID does not exist" containerID="0b831f9a8a9e1f23e4702a9956d942e5cda85b42beef59c94dadb7d458bc6103"
Sep 15 22:46:27.894514 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.894435    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b831f9a8a9e1f23e4702a9956d942e5cda85b42beef59c94dadb7d458bc6103"} err="failed to get container status \"0b831f9a8a9e1f23e4702a9956d942e5cda85b42beef59c94dadb7d458bc6103\": rpc error: code = NotFound desc = could not find container \"0b831f9a8a9e1f23e4702a9956d942e5cda85b42beef59c94dadb7d458bc6103\": container with ID starting with 0b831f9a8a9e1f23e4702a9956d942e5cda85b42beef59c94dadb7d458bc6103 not found: ID does not exist"
Sep 15 22:46:27.911793 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.911770    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"]
Sep 15 22:46:27.917948 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:27.917928    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-4ccd57/explicit-model-4ccd57-kserve-777895cd4f-52qwn"]
Sep 15 22:46:28.114904 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:28.114875    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:28.213058 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:28.212962    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:28.312655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:28.312624    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:28.445400 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:28.445366    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:28.540337 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:28.540203    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:28.978042 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:28.978006    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:29.559159 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:29.559125    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:29.659382 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:29.659348    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:29.805217 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:29.805174    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:29.806622 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:29.806595    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"]
Sep 15 22:46:29.806866 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:29.806841    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerName="istio-proxy" containerID="cri-o://791ac3e1a8913f2da03305a8dcf461151aa20b0fa479b6109fece7202f5a1abc" gracePeriod=30
Sep 15 22:46:29.881066 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:29.880984    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:29.899425 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:29.899399    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6786229b-290f-4a44-b51f-bc654aed9a99" path="/var/lib/kubelet/pods/6786229b-290f-4a44-b51f-bc654aed9a99/volumes"
Sep 15 22:46:30.253698 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:30.253579    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:30.548757 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:30.548718    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:30.617416 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:30.617387    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:31.106742 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:31.106708    2584 patch_prober.go:28] interesting pod/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 22:46:31.106913 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:31.106764    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 22:46:32.365046 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:32.364982    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:32.460342 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:32.460309    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:32.569458 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:32.569425    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:34.930211 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:34.930185    2584 generic.go:356] "Generic (PLEG): container finished" podID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerID="791ac3e1a8913f2da03305a8dcf461151aa20b0fa479b6109fece7202f5a1abc" exitCode=0
Sep 15 22:46:34.930563 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:34.930255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" event={"ID":"35f7985a-eec4-435f-8b3d-786a9198cbce","Type":"ContainerDied","Data":"791ac3e1a8913f2da03305a8dcf461151aa20b0fa479b6109fece7202f5a1abc"}
Sep 15 22:46:34.930563 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:34.930337    2584 scope.go:117] "RemoveContainer" containerID="96852955e3dbe97ded30e00f2ca6439e3ad0ca20b8ab7f03add74db4369486be"
Sep 15 22:46:35.074373 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.074320    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:46:35.136315 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.136287    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-workload-certs\") pod \"35f7985a-eec4-435f-8b3d-786a9198cbce\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") "
Sep 15 22:46:35.136472 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.136337    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-workload-socket\") pod \"35f7985a-eec4-435f-8b3d-786a9198cbce\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") "
Sep 15 22:46:35.136472 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.136372    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-token\") pod \"35f7985a-eec4-435f-8b3d-786a9198cbce\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") "
Sep 15 22:46:35.136472 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.136432    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-envoy\") pod \"35f7985a-eec4-435f-8b3d-786a9198cbce\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") "
Sep 15 22:46:35.136472 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.136461    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9566\" (UniqueName: \"kubernetes.io/projected/35f7985a-eec4-435f-8b3d-786a9198cbce-kube-api-access-q9566\") pod \"35f7985a-eec4-435f-8b3d-786a9198cbce\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") "
Sep 15 22:46:35.136693 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.136495    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-podinfo\") pod \"35f7985a-eec4-435f-8b3d-786a9198cbce\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") "
Sep 15 22:46:35.136693 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.136559    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "35f7985a-eec4-435f-8b3d-786a9198cbce" (UID: "35f7985a-eec4-435f-8b3d-786a9198cbce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:46:35.136693 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.136656    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "35f7985a-eec4-435f-8b3d-786a9198cbce" (UID: "35f7985a-eec4-435f-8b3d-786a9198cbce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:46:35.136894 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.136704    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-data\") pod \"35f7985a-eec4-435f-8b3d-786a9198cbce\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") "
Sep 15 22:46:35.136894 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.136756    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f7985a-eec4-435f-8b3d-786a9198cbce-istiod-ca-cert\") pod \"35f7985a-eec4-435f-8b3d-786a9198cbce\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") "
Sep 15 22:46:35.136894 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.136813    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-credential-socket\") pod \"35f7985a-eec4-435f-8b3d-786a9198cbce\" (UID: \"35f7985a-eec4-435f-8b3d-786a9198cbce\") "
Sep 15 22:46:35.137049 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.136897    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-data" (OuterVolumeSpecName: "istio-data") pod "35f7985a-eec4-435f-8b3d-786a9198cbce" (UID: "35f7985a-eec4-435f-8b3d-786a9198cbce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:46:35.137128 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.137099    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "35f7985a-eec4-435f-8b3d-786a9198cbce" (UID: "35f7985a-eec4-435f-8b3d-786a9198cbce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:46:35.137194 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.137155    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f7985a-eec4-435f-8b3d-786a9198cbce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "35f7985a-eec4-435f-8b3d-786a9198cbce" (UID: "35f7985a-eec4-435f-8b3d-786a9198cbce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:46:35.137259 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.137239    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-data\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:35.137333 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.137278    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-credential-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:35.137333 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.137297    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-workload-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:35.137333 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.137312    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-workload-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:35.138584 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.138557    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "35f7985a-eec4-435f-8b3d-786a9198cbce" (UID: "35f7985a-eec4-435f-8b3d-786a9198cbce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:46:35.138763 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.138744    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f7985a-eec4-435f-8b3d-786a9198cbce-kube-api-access-q9566" (OuterVolumeSpecName: "kube-api-access-q9566") pod "35f7985a-eec4-435f-8b3d-786a9198cbce" (UID: "35f7985a-eec4-435f-8b3d-786a9198cbce"). InnerVolumeSpecName "kube-api-access-q9566". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:46:35.138941 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.138920    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-token" (OuterVolumeSpecName: "istio-token") pod "35f7985a-eec4-435f-8b3d-786a9198cbce" (UID: "35f7985a-eec4-435f-8b3d-786a9198cbce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:46:35.138941 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.138925    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "35f7985a-eec4-435f-8b3d-786a9198cbce" (UID: "35f7985a-eec4-435f-8b3d-786a9198cbce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 22:46:35.238708 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.238682    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-token\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:35.238708 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.238706    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-envoy\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:35.238881 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.238716    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9566\" (UniqueName: \"kubernetes.io/projected/35f7985a-eec4-435f-8b3d-786a9198cbce-kube-api-access-q9566\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:35.238881 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.238727    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f7985a-eec4-435f-8b3d-786a9198cbce-istio-podinfo\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:35.238881 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.238735    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f7985a-eec4-435f-8b3d-786a9198cbce-istiod-ca-cert\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:46:35.937155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.937119    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"
Sep 15 22:46:35.937574 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.937174    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn" event={"ID":"35f7985a-eec4-435f-8b3d-786a9198cbce","Type":"ContainerDied","Data":"95f48fcd39a3a49ac24284b638cd800b31ff9a9c07fe53c0b62e3845d231acbd"}
Sep 15 22:46:35.937574 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.937241    2584 scope.go:117] "RemoveContainer" containerID="791ac3e1a8913f2da03305a8dcf461151aa20b0fa479b6109fece7202f5a1abc"
Sep 15 22:46:35.956495 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.956467    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"]
Sep 15 22:46:35.962874 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:35.962848    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-4ccd57-openshift-default-7f4985dd4-lzddn"]
Sep 15 22:46:37.900671 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:37.900633    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" path="/var/lib/kubelet/pods/35f7985a-eec4-435f-8b3d-786a9198cbce/volumes"
Sep 15 22:46:38.365725 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:38.365693    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:46:50.027698 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.027610    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:50.859999 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.859966    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"]
Sep 15 22:46:50.860471 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.860456    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerName="istio-proxy"
Sep 15 22:46:50.860523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.860472    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerName="istio-proxy"
Sep 15 22:46:50.860523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.860480    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6786229b-290f-4a44-b51f-bc654aed9a99" containerName="main"
Sep 15 22:46:50.860523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.860486    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6786229b-290f-4a44-b51f-bc654aed9a99" containerName="main"
Sep 15 22:46:50.860523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.860501    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerName="istio-proxy"
Sep 15 22:46:50.860523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.860509    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerName="istio-proxy"
Sep 15 22:46:50.860523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.860518    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8bb31a-060e-445d-a626-ed98ed30f289" containerName="revoke-keys"
Sep 15 22:46:50.860523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.860524    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8bb31a-060e-445d-a626-ed98ed30f289" containerName="revoke-keys"
Sep 15 22:46:50.860743 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.860610    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerName="istio-proxy"
Sep 15 22:46:50.860743 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.860619    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="35f7985a-eec4-435f-8b3d-786a9198cbce" containerName="istio-proxy"
Sep 15 22:46:50.860743 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.860628    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6786229b-290f-4a44-b51f-bc654aed9a99" containerName="main"
Sep 15 22:46:50.860743 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.860638    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee8bb31a-060e-445d-a626-ed98ed30f289" containerName="revoke-keys"
Sep 15 22:46:50.865084 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.865065    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:50.869005 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.868965    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-03f258\"/\"e2e-conflict-a-51df66af-kserve-self-signed-certs\""
Sep 15 22:46:50.871092 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:50.871067    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"]
Sep 15 22:46:51.001993 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.001955    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-874bd\" (UniqueName: \"kubernetes.io/projected/65d92989-df52-4f5d-b2d0-899af4dcc946-kube-api-access-874bd\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.002183 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.002062    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-home\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.002183 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.002082    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-dshm\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.002183 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.002106    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-model-cache\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.002183 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.002170    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-tmp-dir\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.002450 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.002188    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d92989-df52-4f5d-b2d0-899af4dcc946-tls-certs\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.103009 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.102977    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-home\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.103009 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.103009    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-dshm\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.103507 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.103035    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-model-cache\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.103507 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.103075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-tmp-dir\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.103507 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.103090    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d92989-df52-4f5d-b2d0-899af4dcc946-tls-certs\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.103507 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.103112    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-874bd\" (UniqueName: \"kubernetes.io/projected/65d92989-df52-4f5d-b2d0-899af4dcc946-kube-api-access-874bd\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.103507 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.103413    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-home\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.103507 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.103442    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-tmp-dir\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.103769 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.103522    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-model-cache\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.105227 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.105204    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-dshm\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.105742 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.105717    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d92989-df52-4f5d-b2d0-899af4dcc946-tls-certs\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.111926 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.111873    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-874bd\" (UniqueName: \"kubernetes.io/projected/65d92989-df52-4f5d-b2d0-899af4dcc946-kube-api-access-874bd\") pod \"e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") " pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.177917 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.177885    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:46:51.450644 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.450620    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"]
Sep 15 22:46:51.452353 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:46:51.452316    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65d92989_df52_4f5d_b2d0_899af4dcc946.slice/crio-c4c899dd1ee69b1da19a61586836f1d0b707d06f7b1703ace7a6123095c3751c WatchSource:0}: Error finding container c4c899dd1ee69b1da19a61586836f1d0b707d06f7b1703ace7a6123095c3751c: Status 404 returned error can't find the container with id c4c899dd1ee69b1da19a61586836f1d0b707d06f7b1703ace7a6123095c3751c
Sep 15 22:46:51.533297 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.533248    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:51.734241 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:51.734146    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:52.036045 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:52.035948    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb" event={"ID":"65d92989-df52-4f5d-b2d0-899af4dcc946","Type":"ContainerStarted","Data":"600ca232ace596ad4d94a28a6fb23b93c073e4c6d8c13dda0e121e4d5b2d5299"}
Sep 15 22:46:52.036045 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:52.035989    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb" event={"ID":"65d92989-df52-4f5d-b2d0-899af4dcc946","Type":"ContainerStarted","Data":"c4c899dd1ee69b1da19a61586836f1d0b707d06f7b1703ace7a6123095c3751c"}
Sep 15 22:46:52.053569 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:52.053524    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb" podStartSLOduration=2.05350879 podStartE2EDuration="2.05350879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:46:52.050801368 +0000 UTC m=+1184.669616277" watchObservedRunningTime="2026-09-15 22:46:52.05350879 +0000 UTC m=+1184.672323745"
Sep 15 22:46:52.406226 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:52.406182    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:53.161651 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:53.161613    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:53.858620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:53.858577    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:54.020773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:54.020741    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:46:54.664587 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:46:54.664552    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:01.178168 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:01.178130    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:47:01.178591 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:01.178207    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:47:01.190785 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:01.190762    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:47:02.110050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:02.110018    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:47:05.230189 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.230153    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:05.499232 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.499150    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"]
Sep 15 22:47:05.503169 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.503149    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.505950 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.505926    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6cdcda22-openshift-default-dockercfg-fbkn9\""
Sep 15 22:47:05.514761 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.514733    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"]
Sep 15 22:47:05.524732 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.524698    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:05.546327 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.546299    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-credential-socket\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.546429 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.546352    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-workload-socket\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.546429 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.546379    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d601f3-2361-4855-8090-643aa8582094-istiod-ca-cert\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.546429 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.546403    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwf7w\" (UniqueName: \"kubernetes.io/projected/40d601f3-2361-4855-8090-643aa8582094-kube-api-access-mwf7w\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.546576 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.546448    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d601f3-2361-4855-8090-643aa8582094-istio-token\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.546576 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.546477    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-workload-certs\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.546695 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.546581    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-istio-data\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.546695 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.546637    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d601f3-2361-4855-8090-643aa8582094-istio-podinfo\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.546695 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.546668    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-istio-envoy\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.647923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.647879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-istio-data\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.648121 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.647948    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d601f3-2361-4855-8090-643aa8582094-istio-podinfo\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.648121 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.648065    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-istio-envoy\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.648254 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.648151    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-credential-socket\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.648254 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.648185    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-workload-socket\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.648254 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.648200    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d601f3-2361-4855-8090-643aa8582094-istiod-ca-cert\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.648254 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.648226    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwf7w\" (UniqueName: \"kubernetes.io/projected/40d601f3-2361-4855-8090-643aa8582094-kube-api-access-mwf7w\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.648506 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.648310    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d601f3-2361-4855-8090-643aa8582094-istio-token\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.648506 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.648350    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-workload-certs\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.648506 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.648399    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-istio-data\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.648653 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.648523    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-credential-socket\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.648725 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.648703    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-workload-certs\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.648972 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.648949    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d601f3-2361-4855-8090-643aa8582094-istiod-ca-cert\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.649090 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.649066    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-workload-socket\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.650523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.650504    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-istio-envoy\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.650925 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.650901    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d601f3-2361-4855-8090-643aa8582094-istio-podinfo\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.656204 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.656175    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d601f3-2361-4855-8090-643aa8582094-istio-token\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.656331 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.656286    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwf7w\" (UniqueName: \"kubernetes.io/projected/40d601f3-2361-4855-8090-643aa8582094-kube-api-access-mwf7w\") pod \"e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") " pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:05.814857 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:05.814826    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:06.011578 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:06.011543    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:06.095206 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:06.095178    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"]
Sep 15 22:47:06.096642 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:47:06.096614    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d601f3_2361_4855_8090_643aa8582094.slice/crio-8623b1a0d21febe7b9fbd00e82e88e0c5baf0ab03d214ecca4f78a37c005fc79 WatchSource:0}: Error finding container 8623b1a0d21febe7b9fbd00e82e88e0c5baf0ab03d214ecca4f78a37c005fc79: Status 404 returned error can't find the container with id 8623b1a0d21febe7b9fbd00e82e88e0c5baf0ab03d214ecca4f78a37c005fc79
Sep 15 22:47:06.098912 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:06.098874    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:47:06.099003 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:06.098943    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:47:06.099003 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:06.098971    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:47:06.124701 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:06.124677    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z" event={"ID":"40d601f3-2361-4855-8090-643aa8582094","Type":"ContainerStarted","Data":"8623b1a0d21febe7b9fbd00e82e88e0c5baf0ab03d214ecca4f78a37c005fc79"}
Sep 15 22:47:07.132688 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:07.132651    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z" event={"ID":"40d601f3-2361-4855-8090-643aa8582094","Type":"ContainerStarted","Data":"a8804954121d4ef8498ce4105583843bf52bb372f535d788866e20448df421cd"}
Sep 15 22:47:07.155038 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:07.154986    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z" podStartSLOduration=2.154970771 podStartE2EDuration="2.154970771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:47:07.151986249 +0000 UTC m=+1199.770801156" watchObservedRunningTime="2026-09-15 22:47:07.154970771 +0000 UTC m=+1199.773785678"
Sep 15 22:47:07.815001 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:07.814963    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:07.820743 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:07.820718    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:07.919466 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:07.919427    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:47:07.919753 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:07.919734    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:47:08.138639 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:08.138557    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:08.139433 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:08.139414    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:08.337464 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:08.337427    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:08.388555 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:08.388522    2584 scope.go:117] "RemoveContainer" containerID="ca525e141e6a8f647b5758ea0d6c4f877dedb1c941fe2e83b0864f256de7aa48"
Sep 15 22:47:09.046003 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:09.045951    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:09.142326 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:09.142287    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:09.391535 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:09.391450    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:10.081650 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:10.081616    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:10.956676 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:10.956640    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:11.648159 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:11.648122    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:12.365786 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:12.365750    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:13.174717 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:13.174686    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:13.897522 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:13.897488    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:13.964093 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:13.964039    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:14.351847 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.351804    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:14.450788 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.450751    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:14.489079 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.489047    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"]
Sep 15 22:47:14.493923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.493905    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"
Sep 15 22:47:14.496912 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.496890    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6cdcda22\""
Sep 15 22:47:14.497078 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.497060    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6cdcda22-dockercfg-r56v6\""
Sep 15 22:47:14.503502 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.503482    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"]
Sep 15 22:47:14.526425 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.526395    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:14.527978 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.527958    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"]
Sep 15 22:47:14.531456 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.531442    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"
Sep 15 22:47:14.539094 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.539060    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94e239b-0a13-4d3a-b497-2dbd34631404-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq\" (UID: \"c94e239b-0a13-4d3a-b497-2dbd34631404\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"
Sep 15 22:47:14.539223 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.539202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffz7\" (UniqueName: \"kubernetes.io/projected/c94e239b-0a13-4d3a-b497-2dbd34631404-kube-api-access-6ffz7\") pod \"payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq\" (UID: \"c94e239b-0a13-4d3a-b497-2dbd34631404\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"
Sep 15 22:47:14.541879 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.541856    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"]
Sep 15 22:47:14.640195 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.640106    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94e239b-0a13-4d3a-b497-2dbd34631404-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq\" (UID: \"c94e239b-0a13-4d3a-b497-2dbd34631404\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"
Sep 15 22:47:14.640195 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.640161    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea649cda-4214-472b-8494-c9d851de1cc3-plugins-config-volume\") pod \"payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv\" (UID: \"ea649cda-4214-472b-8494-c9d851de1cc3\") " pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"
Sep 15 22:47:14.640195 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.640184    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6t8f\" (UniqueName: \"kubernetes.io/projected/ea649cda-4214-472b-8494-c9d851de1cc3-kube-api-access-q6t8f\") pod \"payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv\" (UID: \"ea649cda-4214-472b-8494-c9d851de1cc3\") " pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"
Sep 15 22:47:14.640466 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.640236    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffz7\" (UniqueName: \"kubernetes.io/projected/c94e239b-0a13-4d3a-b497-2dbd34631404-kube-api-access-6ffz7\") pod \"payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq\" (UID: \"c94e239b-0a13-4d3a-b497-2dbd34631404\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"
Sep 15 22:47:14.640932 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.640913    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94e239b-0a13-4d3a-b497-2dbd34631404-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq\" (UID: \"c94e239b-0a13-4d3a-b497-2dbd34631404\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"
Sep 15 22:47:14.648446 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.648423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffz7\" (UniqueName: \"kubernetes.io/projected/c94e239b-0a13-4d3a-b497-2dbd34631404-kube-api-access-6ffz7\") pod \"payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq\" (UID: \"c94e239b-0a13-4d3a-b497-2dbd34631404\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"
Sep 15 22:47:14.741558 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.741524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea649cda-4214-472b-8494-c9d851de1cc3-plugins-config-volume\") pod \"payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv\" (UID: \"ea649cda-4214-472b-8494-c9d851de1cc3\") " pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"
Sep 15 22:47:14.741710 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.741562    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6t8f\" (UniqueName: \"kubernetes.io/projected/ea649cda-4214-472b-8494-c9d851de1cc3-kube-api-access-q6t8f\") pod \"payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv\" (UID: \"ea649cda-4214-472b-8494-c9d851de1cc3\") " pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"
Sep 15 22:47:14.742211 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.742188    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea649cda-4214-472b-8494-c9d851de1cc3-plugins-config-volume\") pod \"payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv\" (UID: \"ea649cda-4214-472b-8494-c9d851de1cc3\") " pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"
Sep 15 22:47:14.749764 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.749740    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6t8f\" (UniqueName: \"kubernetes.io/projected/ea649cda-4214-472b-8494-c9d851de1cc3-kube-api-access-q6t8f\") pod \"payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv\" (UID: \"ea649cda-4214-472b-8494-c9d851de1cc3\") " pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"
Sep 15 22:47:14.805715 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.805686    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"
Sep 15 22:47:14.842120 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:14.842088    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"
Sep 15 22:47:15.080715 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:15.080689    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"]
Sep 15 22:47:15.081908 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:47:15.081879    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94e239b_0a13_4d3a_b497_2dbd34631404.slice/crio-9642288b3ab225432944c36373273cb618d12aa184285c75bcc275cf2d857f2e WatchSource:0}: Error finding container 9642288b3ab225432944c36373273cb618d12aa184285c75bcc275cf2d857f2e: Status 404 returned error can't find the container with id 9642288b3ab225432944c36373273cb618d12aa184285c75bcc275cf2d857f2e
Sep 15 22:47:15.142960 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:15.142938    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"]
Sep 15 22:47:15.144861 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:47:15.144831    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea649cda_4214_472b_8494_c9d851de1cc3.slice/crio-8c5f2fdcdacc6af71bdb44c9d557e0142ca443e70fdb24669a219e3d3a06bae7 WatchSource:0}: Error finding container 8c5f2fdcdacc6af71bdb44c9d557e0142ca443e70fdb24669a219e3d3a06bae7: Status 404 returned error can't find the container with id 8c5f2fdcdacc6af71bdb44c9d557e0142ca443e70fdb24669a219e3d3a06bae7
Sep 15 22:47:15.185815 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:15.185783    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq" event={"ID":"c94e239b-0a13-4d3a-b497-2dbd34631404","Type":"ContainerStarted","Data":"9ffa0b826975f04934059cd9d34ebaba6331f07bfb1bd9e9210a1324c7b3abe7"}
Sep 15 22:47:15.185926 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:15.185856    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq" event={"ID":"c94e239b-0a13-4d3a-b497-2dbd34631404","Type":"ContainerStarted","Data":"9642288b3ab225432944c36373273cb618d12aa184285c75bcc275cf2d857f2e"}
Sep 15 22:47:15.186050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:15.186027    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"
Sep 15 22:47:15.187250 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:15.187226    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv" event={"ID":"ea649cda-4214-472b-8494-c9d851de1cc3","Type":"ContainerStarted","Data":"8c5f2fdcdacc6af71bdb44c9d557e0142ca443e70fdb24669a219e3d3a06bae7"}
Sep 15 22:47:15.203465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:15.203425    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq" podStartSLOduration=1.203407933 podStartE2EDuration="1.203407933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:47:15.202366676 +0000 UTC m=+1207.821181596" watchObservedRunningTime="2026-09-15 22:47:15.203407933 +0000 UTC m=+1207.822222843"
Sep 15 22:47:15.363783 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:15.363749    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:47:16.196727 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:16.196677    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv" event={"ID":"ea649cda-4214-472b-8494-c9d851de1cc3","Type":"ContainerStarted","Data":"c0f6be97b50dc79345222175d5a0459b58e2ba8909d04b89b97704693165d2a1"}
Sep 15 22:47:16.197212 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:16.197116    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"
Sep 15 22:47:16.217863 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:16.217806    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv" podStartSLOduration=2.217785832 podStartE2EDuration="2.217785832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:47:16.213325708 +0000 UTC m=+1208.832140617" watchObservedRunningTime="2026-09-15 22:47:16.217785832 +0000 UTC m=+1208.836600742"
Sep 15 22:47:23.160218 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:23.160188    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:26.199116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:26.199076    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"
Sep 15 22:47:28.211974 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:28.211944    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"
Sep 15 22:47:36.152931 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:36.152886    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:36.969010 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:36.968973    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:37.899035 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:37.899002    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:38.690260 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:38.690227    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:39.078870 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:39.078838    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:39.635515 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:39.635478    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:40.471077 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:40.470996    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:41.294142 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:41.294107    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:42.089077 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:42.089041    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:43.070127 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:43.070086    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:43.360393 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:43.360238    2584 generic.go:356] "Generic (PLEG): container finished" podID="40d601f3-2361-4855-8090-643aa8582094" containerID="a8804954121d4ef8498ce4105583843bf52bb372f535d788866e20448df421cd" exitCode=0
Sep 15 22:47:43.360393 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:43.360311    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z" event={"ID":"40d601f3-2361-4855-8090-643aa8582094","Type":"ContainerDied","Data":"a8804954121d4ef8498ce4105583843bf52bb372f535d788866e20448df421cd"}
Sep 15 22:47:43.360995 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:43.360975    2584 scope.go:117] "RemoveContainer" containerID="a8804954121d4ef8498ce4105583843bf52bb372f535d788866e20448df421cd"
Sep 15 22:47:43.362388 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:43.362361    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:47:43.362463 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:43.362428    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:47:43.362509 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:43.362462    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:47:44.368043 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:44.368011    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z" event={"ID":"40d601f3-2361-4855-8090-643aa8582094","Type":"ContainerStarted","Data":"c69a9b8174db96c58ad145b6eee1864d60549d45753c682a2f367446fb45e648"}
Sep 15 22:47:44.815289 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:44.815234    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:44.815572 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:44.815549    2584 patch_prober.go:28] interesting pod/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 15 22:47:44.815668 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:44.815607    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z" podUID="40d601f3-2361-4855-8090-643aa8582094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 15 22:47:45.816166 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:45.816129    2584 patch_prober.go:28] interesting pod/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 15 22:47:45.816599 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:45.816195    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z" podUID="40d601f3-2361-4855-8090-643aa8582094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 15 22:47:46.815193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:46.815157    2584 patch_prober.go:28] interesting pod/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 15 22:47:46.815385 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:46.815221    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z" podUID="40d601f3-2361-4855-8090-643aa8582094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 15 22:47:47.815183 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:47.815142    2584 patch_prober.go:28] interesting pod/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 15 22:47:47.815595 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:47.815201    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z" podUID="40d601f3-2361-4855-8090-643aa8582094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 15 22:47:48.815553 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:48.815518    2584 patch_prober.go:28] interesting pod/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 15 22:47:48.815940 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:48.815577    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z" podUID="40d601f3-2361-4855-8090-643aa8582094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 15 22:47:49.819933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:49.819900    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:49.820298 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:49.820283    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:49.820962 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:49.820947    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:47:50.475403 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:50.475372    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:47:55.546455 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:55.546417    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:55.755106 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:55.755068    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:55.855644 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:55.855562    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:56.543834 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.543797    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"]
Sep 15 22:47:56.548961 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.548938    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.551872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.551849    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fe020d\"/\"kube-root-ca.crt\""
Sep 15 22:47:56.553113 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.553091    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fe020d\"/\"distinct-w0-fe020d-kserve-self-signed-certs\""
Sep 15 22:47:56.553218 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.553142    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fe020d\"/\"openshift-service-ca.crt\""
Sep 15 22:47:56.553218 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.553179    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fe020d\"/\"default-dockercfg-t99pp\""
Sep 15 22:47:56.558985 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.558963    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"]
Sep 15 22:47:56.648875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.648845    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twngd\" (UniqueName: \"kubernetes.io/projected/524703bc-7a31-4028-b790-543f88a6dcd9-kube-api-access-twngd\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.649016 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.648924    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-tmp-dir\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.649016 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.648960    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-home\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.649016 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.648992    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/524703bc-7a31-4028-b790-543f88a6dcd9-tls-certs\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.649185 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.649066    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-dshm\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.649185 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.649096    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-model-cache\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.750049 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.750023    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-dshm\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.750202 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.750057    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-model-cache\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.750202 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.750100    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twngd\" (UniqueName: \"kubernetes.io/projected/524703bc-7a31-4028-b790-543f88a6dcd9-kube-api-access-twngd\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.750202 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.750170    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-tmp-dir\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.750420 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.750206    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-home\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.750420 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.750234    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/524703bc-7a31-4028-b790-543f88a6dcd9-tls-certs\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.750543 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.750508    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-model-cache\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.750586 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.750547    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-tmp-dir\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.750622 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.750602    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-home\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.752386 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.752364    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-dshm\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.752650 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.752632    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/524703bc-7a31-4028-b790-543f88a6dcd9-tls-certs\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.758506 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.758484    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twngd\" (UniqueName: \"kubernetes.io/projected/524703bc-7a31-4028-b790-543f88a6dcd9-kube-api-access-twngd\") pod \"distinct-w0-fe020d-kserve-b64c474d8-4x7w6\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:56.844678 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.844648    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:56.859604 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:56.859587    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:47:57.139144 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:57.139105    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"]
Sep 15 22:47:57.140968 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:47:57.140927    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod524703bc_7a31_4028_b790_543f88a6dcd9.slice/crio-21ba4cec8d06dc3da8217e7dbd9a59bdc3021f2c7671e0fe9a405c01a17ae92e WatchSource:0}: Error finding container 21ba4cec8d06dc3da8217e7dbd9a59bdc3021f2c7671e0fe9a405c01a17ae92e: Status 404 returned error can't find the container with id 21ba4cec8d06dc3da8217e7dbd9a59bdc3021f2c7671e0fe9a405c01a17ae92e
Sep 15 22:47:57.454807 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:57.454712    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6" event={"ID":"524703bc-7a31-4028-b790-543f88a6dcd9","Type":"ContainerStarted","Data":"28bbfbc95745484411e62a5d1398b72c453c1855f4bd983747462d076f029eed"}
Sep 15 22:47:57.454807 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:57.454764    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6" event={"ID":"524703bc-7a31-4028-b790-543f88a6dcd9","Type":"ContainerStarted","Data":"21ba4cec8d06dc3da8217e7dbd9a59bdc3021f2c7671e0fe9a405c01a17ae92e"}
Sep 15 22:47:57.476286 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:57.474962    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6" podStartSLOduration=1.4749386279999999 podStartE2EDuration="1.474938628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:47:57.47069632 +0000 UTC m=+1250.089511228" watchObservedRunningTime="2026-09-15 22:47:57.474938628 +0000 UTC m=+1250.093753537"
Sep 15 22:47:59.318643 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:59.318606    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4"]
Sep 15 22:47:59.322431 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:59.322413    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4"
Sep 15 22:47:59.325312 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:59.325283    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:47:59.325312 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:59.325290    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:47:59.330677 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:59.330657    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4"]
Sep 15 22:47:59.373248 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:59.373211    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9270e7-fe87-4feb-9b8a-cd10eecc0d35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4\" (UID: \"ac9270e7-fe87-4feb-9b8a-cd10eecc0d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4"
Sep 15 22:47:59.474716 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:59.474687    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9270e7-fe87-4feb-9b8a-cd10eecc0d35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4\" (UID: \"ac9270e7-fe87-4feb-9b8a-cd10eecc0d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4"
Sep 15 22:47:59.475373 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:59.475348    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9270e7-fe87-4feb-9b8a-cd10eecc0d35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4\" (UID: \"ac9270e7-fe87-4feb-9b8a-cd10eecc0d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4"
Sep 15 22:47:59.585236 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:59.585151    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:47:59.634433 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:59.634406    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4"
Sep 15 22:47:59.915283 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:47:59.915056    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4"]
Sep 15 22:47:59.918106 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:47:59.918080    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9270e7_fe87_4feb_9b8a_cd10eecc0d35.slice/crio-4e720586c0987452eec2024b00d91826287e4f19cdca4aa370d2670778f54084 WatchSource:0}: Error finding container 4e720586c0987452eec2024b00d91826287e4f19cdca4aa370d2670778f54084: Status 404 returned error can't find the container with id 4e720586c0987452eec2024b00d91826287e4f19cdca4aa370d2670778f54084
Sep 15 22:48:00.382101 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:00.382066    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:00.474974 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:00.474880    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4" event={"ID":"ac9270e7-fe87-4feb-9b8a-cd10eecc0d35","Type":"ContainerStarted","Data":"1a1e2824716e61a5769e94aa7a14343f2d1e0163eff074d0f9428a75d8991d54"}
Sep 15 22:48:00.474974 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:00.474937    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4" event={"ID":"ac9270e7-fe87-4feb-9b8a-cd10eecc0d35","Type":"ContainerStarted","Data":"4e720586c0987452eec2024b00d91826287e4f19cdca4aa370d2670778f54084"}
Sep 15 22:48:00.492814 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:00.492769    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4" podStartSLOduration=1.492754096 podStartE2EDuration="1.492754096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:48:00.490173862 +0000 UTC m=+1253.108988774" watchObservedRunningTime="2026-09-15 22:48:00.492754096 +0000 UTC m=+1253.111569004"
Sep 15 22:48:00.891996 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:00.891933    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:01.079121 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:01.079085    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:01.482936 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:01.482886    2584 generic.go:356] "Generic (PLEG): container finished" podID="ac9270e7-fe87-4feb-9b8a-cd10eecc0d35" containerID="1a1e2824716e61a5769e94aa7a14343f2d1e0163eff074d0f9428a75d8991d54" exitCode=0
Sep 15 22:48:01.483356 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:01.482955    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4" event={"ID":"ac9270e7-fe87-4feb-9b8a-cd10eecc0d35","Type":"ContainerDied","Data":"1a1e2824716e61a5769e94aa7a14343f2d1e0163eff074d0f9428a75d8991d54"}
Sep 15 22:48:01.786999 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:01.786877    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:02.578661 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:02.578624    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:02.624655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:02.624631    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4"
Sep 15 22:48:02.708591 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:02.708562    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9270e7-fe87-4feb-9b8a-cd10eecc0d35-maas-api-service-ca\") pod \"ac9270e7-fe87-4feb-9b8a-cd10eecc0d35\" (UID: \"ac9270e7-fe87-4feb-9b8a-cd10eecc0d35\") "
Sep 15 22:48:02.708939 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:02.708912    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9270e7-fe87-4feb-9b8a-cd10eecc0d35-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac9270e7-fe87-4feb-9b8a-cd10eecc0d35" (UID: "ac9270e7-fe87-4feb-9b8a-cd10eecc0d35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:48:02.810390 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:02.810365    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9270e7-fe87-4feb-9b8a-cd10eecc0d35-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:48:03.282453 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:03.282371    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:03.496610 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:03.496584    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4"
Sep 15 22:48:03.496799 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:03.496580    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4" event={"ID":"ac9270e7-fe87-4feb-9b8a-cd10eecc0d35","Type":"ContainerDied","Data":"4e720586c0987452eec2024b00d91826287e4f19cdca4aa370d2670778f54084"}
Sep 15 22:48:03.496799 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:03.496717    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e720586c0987452eec2024b00d91826287e4f19cdca4aa370d2670778f54084"
Sep 15 22:48:04.194561 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:04.194517    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:05.003607 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:05.003570    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:05.821877 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:05.821831    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:06.860593 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:06.860560    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:48:06.860593 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:06.860601    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:48:06.872823 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:06.872799    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:48:07.531899 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:07.531869    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:48:08.424110 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:08.424078    2584 scope.go:117] "RemoveContainer" containerID="638acfcc818a7b936f0635c2ddfef27876e81e521c8f62435ca0280d6acaa793"
Sep 15 22:48:09.824061 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:09.824022    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz"]
Sep 15 22:48:09.828245 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:09.828198    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:09.829028 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:09.829005    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fffdae-29825205-v66zz"]
Sep 15 22:48:09.847477 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:09.847446    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"]
Sep 15 22:48:09.847742 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:09.847691    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j" podUID="922688b5-15dc-40c6-a4b1-be53e2ba7881" containerName="payload-processing" containerID="cri-o://69fa88e36c65ba85a4840311402b2ca736b33cdc63c7ea090560a08fc31727e2" gracePeriod=30
Sep 15 22:48:09.891849 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:09.891808    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:09.901745 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:09.901715    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba140266-cadf-4064-88ad-b062eaf06a6c" path="/var/lib/kubelet/pods/ba140266-cadf-4064-88ad-b062eaf06a6c/volumes"
Sep 15 22:48:09.993069 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:09.993034    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:10.021380 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.021341    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4"]
Sep 15 22:48:10.025338 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.025312    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fffdae-rtmd4"]
Sep 15 22:48:10.048614 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.048573    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:10.124393 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.124365    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"
Sep 15 22:48:10.185863 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.185832    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922688b5-15dc-40c6-a4b1-be53e2ba7881-plugins-config-volume\") pod \"922688b5-15dc-40c6-a4b1-be53e2ba7881\" (UID: \"922688b5-15dc-40c6-a4b1-be53e2ba7881\") "
Sep 15 22:48:10.186059 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.185886    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv4sd\" (UniqueName: \"kubernetes.io/projected/922688b5-15dc-40c6-a4b1-be53e2ba7881-kube-api-access-mv4sd\") pod \"922688b5-15dc-40c6-a4b1-be53e2ba7881\" (UID: \"922688b5-15dc-40c6-a4b1-be53e2ba7881\") "
Sep 15 22:48:10.186330 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.186262    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922688b5-15dc-40c6-a4b1-be53e2ba7881-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "922688b5-15dc-40c6-a4b1-be53e2ba7881" (UID: "922688b5-15dc-40c6-a4b1-be53e2ba7881"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:48:10.188298 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.188256    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922688b5-15dc-40c6-a4b1-be53e2ba7881-kube-api-access-mv4sd" (OuterVolumeSpecName: "kube-api-access-mv4sd") pod "922688b5-15dc-40c6-a4b1-be53e2ba7881" (UID: "922688b5-15dc-40c6-a4b1-be53e2ba7881"). InnerVolumeSpecName "kube-api-access-mv4sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:48:10.287714 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.287677    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv4sd\" (UniqueName: \"kubernetes.io/projected/922688b5-15dc-40c6-a4b1-be53e2ba7881-kube-api-access-mv4sd\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:48:10.287714 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.287712    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922688b5-15dc-40c6-a4b1-be53e2ba7881-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:48:10.374097 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.374010    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:10.475739 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.475709    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:10.541328 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.541284    2584 generic.go:356] "Generic (PLEG): container finished" podID="922688b5-15dc-40c6-a4b1-be53e2ba7881" containerID="69fa88e36c65ba85a4840311402b2ca736b33cdc63c7ea090560a08fc31727e2" exitCode=0
Sep 15 22:48:10.541569 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.541384    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j" event={"ID":"922688b5-15dc-40c6-a4b1-be53e2ba7881","Type":"ContainerDied","Data":"69fa88e36c65ba85a4840311402b2ca736b33cdc63c7ea090560a08fc31727e2"}
Sep 15 22:48:10.541569 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.541423    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"
Sep 15 22:48:10.541569 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.541451    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j" event={"ID":"922688b5-15dc-40c6-a4b1-be53e2ba7881","Type":"ContainerDied","Data":"c9a4a79b6c8ccd77d1cbc5dffac089f37599a379db0f5f23c8df45805a1f5184"}
Sep 15 22:48:10.541569 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.541484    2584 scope.go:117] "RemoveContainer" containerID="69fa88e36c65ba85a4840311402b2ca736b33cdc63c7ea090560a08fc31727e2"
Sep 15 22:48:10.553820 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.553801    2584 scope.go:117] "RemoveContainer" containerID="69fa88e36c65ba85a4840311402b2ca736b33cdc63c7ea090560a08fc31727e2"
Sep 15 22:48:10.554091 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:48:10.554072    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69fa88e36c65ba85a4840311402b2ca736b33cdc63c7ea090560a08fc31727e2\": container with ID starting with 69fa88e36c65ba85a4840311402b2ca736b33cdc63c7ea090560a08fc31727e2 not found: ID does not exist" containerID="69fa88e36c65ba85a4840311402b2ca736b33cdc63c7ea090560a08fc31727e2"
Sep 15 22:48:10.554149 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.554100    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69fa88e36c65ba85a4840311402b2ca736b33cdc63c7ea090560a08fc31727e2"} err="failed to get container status \"69fa88e36c65ba85a4840311402b2ca736b33cdc63c7ea090560a08fc31727e2\": rpc error: code = NotFound desc = could not find container \"69fa88e36c65ba85a4840311402b2ca736b33cdc63c7ea090560a08fc31727e2\": container with ID starting with 69fa88e36c65ba85a4840311402b2ca736b33cdc63c7ea090560a08fc31727e2 not found: ID does not exist"
Sep 15 22:48:10.567416 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.567392    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"]
Sep 15 22:48:10.571183 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:10.571160    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fffdae-7f945576fc-z9j7j"]
Sep 15 22:48:11.194053 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:11.194007    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:11.901537 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:11.901494    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="922688b5-15dc-40c6-a4b1-be53e2ba7881" path="/var/lib/kubelet/pods/922688b5-15dc-40c6-a4b1-be53e2ba7881/volumes"
Sep 15 22:48:11.902003 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:11.901984    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac9270e7-fe87-4feb-9b8a-cd10eecc0d35" path="/var/lib/kubelet/pods/ac9270e7-fe87-4feb-9b8a-cd10eecc0d35/volumes"
Sep 15 22:48:12.056735 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:12.056705    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:12.858776 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:12.858741    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:13.663013 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:13.662977    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:14.478097 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:14.478036    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:15.256299 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:15.256243    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:15.457529 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:15.457499    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:15.841654 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:15.841619    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:17.830083 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:17.830048    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:18.335200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:18.335159    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"]
Sep 15 22:48:18.359530 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:18.359501    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:25.663155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:25.663114    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:48:28.384196 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:28.384156    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:33.355696 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.355654    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst" podUID="d86ff26d-da6e-4314-9330-092e2e50e17b" containerName="main" containerID="cri-o://acabb0a94f92e01ba9a49453eaccdf51927cb02be59b38d403b08bc63e303676" gracePeriod=45
Sep 15 22:48:33.625254 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.625225    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:48:33.686738 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.686706    2584 generic.go:356] "Generic (PLEG): container finished" podID="d86ff26d-da6e-4314-9330-092e2e50e17b" containerID="acabb0a94f92e01ba9a49453eaccdf51927cb02be59b38d403b08bc63e303676" exitCode=0
Sep 15 22:48:33.686878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.686748    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"
Sep 15 22:48:33.686878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.686790    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst" event={"ID":"d86ff26d-da6e-4314-9330-092e2e50e17b","Type":"ContainerDied","Data":"acabb0a94f92e01ba9a49453eaccdf51927cb02be59b38d403b08bc63e303676"}
Sep 15 22:48:33.686878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.686833    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst" event={"ID":"d86ff26d-da6e-4314-9330-092e2e50e17b","Type":"ContainerDied","Data":"ac58dd447bca56c24e7f9637fe8118641bc46758e42391b7681635989bbcaca9"}
Sep 15 22:48:33.686878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.686851    2584 scope.go:117] "RemoveContainer" containerID="acabb0a94f92e01ba9a49453eaccdf51927cb02be59b38d403b08bc63e303676"
Sep 15 22:48:33.695750 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.695730    2584 scope.go:117] "RemoveContainer" containerID="acabb0a94f92e01ba9a49453eaccdf51927cb02be59b38d403b08bc63e303676"
Sep 15 22:48:33.696002 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:48:33.695983    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acabb0a94f92e01ba9a49453eaccdf51927cb02be59b38d403b08bc63e303676\": container with ID starting with acabb0a94f92e01ba9a49453eaccdf51927cb02be59b38d403b08bc63e303676 not found: ID does not exist" containerID="acabb0a94f92e01ba9a49453eaccdf51927cb02be59b38d403b08bc63e303676"
Sep 15 22:48:33.696050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.696010    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acabb0a94f92e01ba9a49453eaccdf51927cb02be59b38d403b08bc63e303676"} err="failed to get container status \"acabb0a94f92e01ba9a49453eaccdf51927cb02be59b38d403b08bc63e303676\": rpc error: code = NotFound desc = could not find container \"acabb0a94f92e01ba9a49453eaccdf51927cb02be59b38d403b08bc63e303676\": container with ID starting with acabb0a94f92e01ba9a49453eaccdf51927cb02be59b38d403b08bc63e303676 not found: ID does not exist"
Sep 15 22:48:33.716397 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.716376    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j25gf\" (UniqueName: \"kubernetes.io/projected/d86ff26d-da6e-4314-9330-092e2e50e17b-kube-api-access-j25gf\") pod \"d86ff26d-da6e-4314-9330-092e2e50e17b\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") "
Sep 15 22:48:33.716475 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.716411    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-model-cache\") pod \"d86ff26d-da6e-4314-9330-092e2e50e17b\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") "
Sep 15 22:48:33.716475 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.716433    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-tmp-dir\") pod \"d86ff26d-da6e-4314-9330-092e2e50e17b\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") "
Sep 15 22:48:33.716475 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.716462    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-dshm\") pod \"d86ff26d-da6e-4314-9330-092e2e50e17b\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") "
Sep 15 22:48:33.716626 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.716612    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-home\") pod \"d86ff26d-da6e-4314-9330-092e2e50e17b\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") "
Sep 15 22:48:33.716678 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.716652    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86ff26d-da6e-4314-9330-092e2e50e17b-tls-certs\") pod \"d86ff26d-da6e-4314-9330-092e2e50e17b\" (UID: \"d86ff26d-da6e-4314-9330-092e2e50e17b\") "
Sep 15 22:48:33.716678 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.716665    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d86ff26d-da6e-4314-9330-092e2e50e17b" (UID: "d86ff26d-da6e-4314-9330-092e2e50e17b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:48:33.716759 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.716676    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d86ff26d-da6e-4314-9330-092e2e50e17b" (UID: "d86ff26d-da6e-4314-9330-092e2e50e17b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:48:33.716919 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.716886    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-home" (OuterVolumeSpecName: "home") pod "d86ff26d-da6e-4314-9330-092e2e50e17b" (UID: "d86ff26d-da6e-4314-9330-092e2e50e17b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:48:33.717039 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.717013    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:48:33.717039 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.717036    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:48:33.717201 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.717052    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:48:33.718606 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.718582    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-dshm" (OuterVolumeSpecName: "dshm") pod "d86ff26d-da6e-4314-9330-092e2e50e17b" (UID: "d86ff26d-da6e-4314-9330-092e2e50e17b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:48:33.718710 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.718663    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86ff26d-da6e-4314-9330-092e2e50e17b-kube-api-access-j25gf" (OuterVolumeSpecName: "kube-api-access-j25gf") pod "d86ff26d-da6e-4314-9330-092e2e50e17b" (UID: "d86ff26d-da6e-4314-9330-092e2e50e17b"). InnerVolumeSpecName "kube-api-access-j25gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:48:33.718710 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.718679    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86ff26d-da6e-4314-9330-092e2e50e17b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d86ff26d-da6e-4314-9330-092e2e50e17b" (UID: "d86ff26d-da6e-4314-9330-092e2e50e17b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:48:33.818025 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.818003    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86ff26d-da6e-4314-9330-092e2e50e17b-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:48:33.818025 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.818023    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j25gf\" (UniqueName: \"kubernetes.io/projected/d86ff26d-da6e-4314-9330-092e2e50e17b-kube-api-access-j25gf\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:48:33.818164 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:33.818033    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86ff26d-da6e-4314-9330-092e2e50e17b-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:48:34.002617 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:34.002585    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"]
Sep 15 22:48:34.008596 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:34.008569    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-765e6d/test-model-765e6d-kserve-b5d6f9b7f-trfst"]
Sep 15 22:48:34.652115 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:34.652073    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:35.899463 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:35.899433    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d86ff26d-da6e-4314-9330-092e2e50e17b" path="/var/lib/kubelet/pods/d86ff26d-da6e-4314-9330-092e2e50e17b/volumes"
Sep 15 22:48:57.661552 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:57.661517    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:48:59.296444 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.296407    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:59.524183 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.524149    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"]
Sep 15 22:48:59.524432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.524407    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw" podUID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerName="istio-proxy" containerID="cri-o://fa427d0e2f6b22e4e4be34af6576dd33bbf2c8ecc3e6b81a4c5770d4b6eb6b9a" gracePeriod=30
Sep 15 22:48:59.567157 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.567073    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:48:59.876433 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.876350    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr"]
Sep 15 22:48:59.876929 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.876908    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac9270e7-fe87-4feb-9b8a-cd10eecc0d35" containerName="revoke-keys"
Sep 15 22:48:59.876929 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.876929    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac9270e7-fe87-4feb-9b8a-cd10eecc0d35" containerName="revoke-keys"
Sep 15 22:48:59.877081 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.876958    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922688b5-15dc-40c6-a4b1-be53e2ba7881" containerName="payload-processing"
Sep 15 22:48:59.877081 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.876967    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="922688b5-15dc-40c6-a4b1-be53e2ba7881" containerName="payload-processing"
Sep 15 22:48:59.877081 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.876986    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86ff26d-da6e-4314-9330-092e2e50e17b" containerName="main"
Sep 15 22:48:59.877081 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.876994    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86ff26d-da6e-4314-9330-092e2e50e17b" containerName="main"
Sep 15 22:48:59.877296 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.877099    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d86ff26d-da6e-4314-9330-092e2e50e17b" containerName="main"
Sep 15 22:48:59.877296 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.877114    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac9270e7-fe87-4feb-9b8a-cd10eecc0d35" containerName="revoke-keys"
Sep 15 22:48:59.877296 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.877125    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="922688b5-15dc-40c6-a4b1-be53e2ba7881" containerName="payload-processing"
Sep 15 22:48:59.880576 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.880556    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr"
Sep 15 22:48:59.883384 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.883359    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:48:59.883533 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.883506    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:48:59.888310 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.888285    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr"]
Sep 15 22:48:59.969361 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:48:59.969334    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30fd544-0cf7-4af4-a729-37e2a872b4b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr\" (UID: \"a30fd544-0cf7-4af4-a729-37e2a872b4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr"
Sep 15 22:49:00.070886 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:00.070843    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30fd544-0cf7-4af4-a729-37e2a872b4b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr\" (UID: \"a30fd544-0cf7-4af4-a729-37e2a872b4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr"
Sep 15 22:49:00.071565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:00.071541    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30fd544-0cf7-4af4-a729-37e2a872b4b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr\" (UID: \"a30fd544-0cf7-4af4-a729-37e2a872b4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr"
Sep 15 22:49:00.162302 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:00.162195    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:00.193926 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:00.193899    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr"
Sep 15 22:49:00.472327 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:00.472299    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr"]
Sep 15 22:49:00.474010 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:49:00.473987    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30fd544_0cf7_4af4_a729_37e2a872b4b5.slice/crio-3552576f6ab87e5210f5a7d89ed94b5e972a673be775b6bdfe8e7d226da98160 WatchSource:0}: Error finding container 3552576f6ab87e5210f5a7d89ed94b5e972a673be775b6bdfe8e7d226da98160: Status 404 returned error can't find the container with id 3552576f6ab87e5210f5a7d89ed94b5e972a673be775b6bdfe8e7d226da98160
Sep 15 22:49:00.856586 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:00.856548    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr" event={"ID":"a30fd544-0cf7-4af4-a729-37e2a872b4b5","Type":"ContainerStarted","Data":"de4c0042f09ad4b3e786b67f8f942d9bbfe0815503cee7209715a20b96506689"}
Sep 15 22:49:00.856712 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:00.856609    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr" event={"ID":"a30fd544-0cf7-4af4-a729-37e2a872b4b5","Type":"ContainerStarted","Data":"3552576f6ab87e5210f5a7d89ed94b5e972a673be775b6bdfe8e7d226da98160"}
Sep 15 22:49:00.875059 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:00.875004    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr" podStartSLOduration=1.874966299 podStartE2EDuration="1.874966299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:49:00.87246037 +0000 UTC m=+1313.491275278" watchObservedRunningTime="2026-09-15 22:49:00.874966299 +0000 UTC m=+1313.493781207"
Sep 15 22:49:00.876625 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:00.876602    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:01.785938 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:01.785898    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:01.864254 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:01.864220    2584 generic.go:356] "Generic (PLEG): container finished" podID="a30fd544-0cf7-4af4-a729-37e2a872b4b5" containerID="de4c0042f09ad4b3e786b67f8f942d9bbfe0815503cee7209715a20b96506689" exitCode=0
Sep 15 22:49:01.864445 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:01.864315    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr" event={"ID":"a30fd544-0cf7-4af4-a729-37e2a872b4b5","Type":"ContainerDied","Data":"de4c0042f09ad4b3e786b67f8f942d9bbfe0815503cee7209715a20b96506689"}
Sep 15 22:49:02.489200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:02.489165    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:03.016980 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:03.016955    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr"
Sep 15 22:49:03.204351 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:03.204258    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30fd544-0cf7-4af4-a729-37e2a872b4b5-maas-api-service-ca\") pod \"a30fd544-0cf7-4af4-a729-37e2a872b4b5\" (UID: \"a30fd544-0cf7-4af4-a729-37e2a872b4b5\") "
Sep 15 22:49:03.204534 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:03.204512    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30fd544-0cf7-4af4-a729-37e2a872b4b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a30fd544-0cf7-4af4-a729-37e2a872b4b5" (UID: "a30fd544-0cf7-4af4-a729-37e2a872b4b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:49:03.204685 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:03.204672    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30fd544-0cf7-4af4-a729-37e2a872b4b5-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:49:03.241779 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:03.241745    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:03.877425 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:03.877390    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr"
Sep 15 22:49:03.877580 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:03.877418    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr" event={"ID":"a30fd544-0cf7-4af4-a729-37e2a872b4b5","Type":"ContainerDied","Data":"3552576f6ab87e5210f5a7d89ed94b5e972a673be775b6bdfe8e7d226da98160"}
Sep 15 22:49:03.877580 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:03.877473    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3552576f6ab87e5210f5a7d89ed94b5e972a673be775b6bdfe8e7d226da98160"
Sep 15 22:49:03.976438 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:03.976408    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:04.691883 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.691846    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:04.797557 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.797536    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:49:04.884887 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.884852    2584 generic.go:356] "Generic (PLEG): container finished" podID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerID="fa427d0e2f6b22e4e4be34af6576dd33bbf2c8ecc3e6b81a4c5770d4b6eb6b9a" exitCode=0
Sep 15 22:49:04.885090 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.884919    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"
Sep 15 22:49:04.885090 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.884926    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw" event={"ID":"5bda438d-af76-4c2d-8b4e-f39d13c857ea","Type":"ContainerDied","Data":"fa427d0e2f6b22e4e4be34af6576dd33bbf2c8ecc3e6b81a4c5770d4b6eb6b9a"}
Sep 15 22:49:04.885090 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.884973    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw" event={"ID":"5bda438d-af76-4c2d-8b4e-f39d13c857ea","Type":"ContainerDied","Data":"4eb38a0fafe098fbd274855447f0985d5ec1401a693a5c376b2b5a19da56bd6e"}
Sep 15 22:49:04.885090 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.884996    2584 scope.go:117] "RemoveContainer" containerID="fa427d0e2f6b22e4e4be34af6576dd33bbf2c8ecc3e6b81a4c5770d4b6eb6b9a"
Sep 15 22:49:04.896473 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.896454    2584 scope.go:117] "RemoveContainer" containerID="570bbb34c381c0c606cf5a9d6e022dca0e024d4aaaaa9faac05da476d607274b"
Sep 15 22:49:04.905196 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.905162    2584 scope.go:117] "RemoveContainer" containerID="fa427d0e2f6b22e4e4be34af6576dd33bbf2c8ecc3e6b81a4c5770d4b6eb6b9a"
Sep 15 22:49:04.905490 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:49:04.905469    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa427d0e2f6b22e4e4be34af6576dd33bbf2c8ecc3e6b81a4c5770d4b6eb6b9a\": container with ID starting with fa427d0e2f6b22e4e4be34af6576dd33bbf2c8ecc3e6b81a4c5770d4b6eb6b9a not found: ID does not exist" containerID="fa427d0e2f6b22e4e4be34af6576dd33bbf2c8ecc3e6b81a4c5770d4b6eb6b9a"
Sep 15 22:49:04.905565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.905500    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa427d0e2f6b22e4e4be34af6576dd33bbf2c8ecc3e6b81a4c5770d4b6eb6b9a"} err="failed to get container status \"fa427d0e2f6b22e4e4be34af6576dd33bbf2c8ecc3e6b81a4c5770d4b6eb6b9a\": rpc error: code = NotFound desc = could not find container \"fa427d0e2f6b22e4e4be34af6576dd33bbf2c8ecc3e6b81a4c5770d4b6eb6b9a\": container with ID starting with fa427d0e2f6b22e4e4be34af6576dd33bbf2c8ecc3e6b81a4c5770d4b6eb6b9a not found: ID does not exist"
Sep 15 22:49:04.905565 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.905517    2584 scope.go:117] "RemoveContainer" containerID="570bbb34c381c0c606cf5a9d6e022dca0e024d4aaaaa9faac05da476d607274b"
Sep 15 22:49:04.905782 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:49:04.905766    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570bbb34c381c0c606cf5a9d6e022dca0e024d4aaaaa9faac05da476d607274b\": container with ID starting with 570bbb34c381c0c606cf5a9d6e022dca0e024d4aaaaa9faac05da476d607274b not found: ID does not exist" containerID="570bbb34c381c0c606cf5a9d6e022dca0e024d4aaaaa9faac05da476d607274b"
Sep 15 22:49:04.905833 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.905786    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570bbb34c381c0c606cf5a9d6e022dca0e024d4aaaaa9faac05da476d607274b"} err="failed to get container status \"570bbb34c381c0c606cf5a9d6e022dca0e024d4aaaaa9faac05da476d607274b\": rpc error: code = NotFound desc = could not find container \"570bbb34c381c0c606cf5a9d6e022dca0e024d4aaaaa9faac05da476d607274b\": container with ID starting with 570bbb34c381c0c606cf5a9d6e022dca0e024d4aaaaa9faac05da476d607274b not found: ID does not exist"
Sep 15 22:49:04.920172 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920148    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-envoy\") pod \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") "
Sep 15 22:49:04.920242 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920183    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-workload-certs\") pod \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") "
Sep 15 22:49:04.920242 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920210    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2zvk\" (UniqueName: \"kubernetes.io/projected/5bda438d-af76-4c2d-8b4e-f39d13c857ea-kube-api-access-f2zvk\") pod \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") "
Sep 15 22:49:04.920242 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920229    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-data\") pod \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") "
Sep 15 22:49:04.920437 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920252    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-podinfo\") pod \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") "
Sep 15 22:49:04.920437 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920316    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-credential-socket\") pod \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") "
Sep 15 22:49:04.920437 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920349    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-workload-socket\") pod \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") "
Sep 15 22:49:04.920437 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920386    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istiod-ca-cert\") pod \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") "
Sep 15 22:49:04.920437 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920438    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-token\") pod \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\" (UID: \"5bda438d-af76-4c2d-8b4e-f39d13c857ea\") "
Sep 15 22:49:04.920706 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920450    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5bda438d-af76-4c2d-8b4e-f39d13c857ea" (UID: "5bda438d-af76-4c2d-8b4e-f39d13c857ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:49:04.920706 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920643    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5bda438d-af76-4c2d-8b4e-f39d13c857ea" (UID: "5bda438d-af76-4c2d-8b4e-f39d13c857ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:49:04.920706 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920660    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5bda438d-af76-4c2d-8b4e-f39d13c857ea" (UID: "5bda438d-af76-4c2d-8b4e-f39d13c857ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:49:04.920861 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920757    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-data" (OuterVolumeSpecName: "istio-data") pod "5bda438d-af76-4c2d-8b4e-f39d13c857ea" (UID: "5bda438d-af76-4c2d-8b4e-f39d13c857ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:49:04.920918 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920863    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-workload-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:49:04.920918 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920885    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-data\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:49:04.920918 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920899    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-credential-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:49:04.920918 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920914    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-workload-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:49:04.921126 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.920866    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5bda438d-af76-4c2d-8b4e-f39d13c857ea" (UID: "5bda438d-af76-4c2d-8b4e-f39d13c857ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:49:04.922459 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.922434    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5bda438d-af76-4c2d-8b4e-f39d13c857ea" (UID: "5bda438d-af76-4c2d-8b4e-f39d13c857ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 22:49:04.922562 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.922533    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bda438d-af76-4c2d-8b4e-f39d13c857ea-kube-api-access-f2zvk" (OuterVolumeSpecName: "kube-api-access-f2zvk") pod "5bda438d-af76-4c2d-8b4e-f39d13c857ea" (UID: "5bda438d-af76-4c2d-8b4e-f39d13c857ea"). InnerVolumeSpecName "kube-api-access-f2zvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:49:04.922613 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.922558    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5bda438d-af76-4c2d-8b4e-f39d13c857ea" (UID: "5bda438d-af76-4c2d-8b4e-f39d13c857ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:49:04.922663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:04.922639    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "5bda438d-af76-4c2d-8b4e-f39d13c857ea" (UID: "5bda438d-af76-4c2d-8b4e-f39d13c857ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:49:05.022324 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:05.022291    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-envoy\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:49:05.022324 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:05.022320    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2zvk\" (UniqueName: \"kubernetes.io/projected/5bda438d-af76-4c2d-8b4e-f39d13c857ea-kube-api-access-f2zvk\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:49:05.022533 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:05.022334    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-podinfo\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:49:05.022533 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:05.022346    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istiod-ca-cert\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:49:05.022533 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:05.022355    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bda438d-af76-4c2d-8b4e-f39d13c857ea-istio-token\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:49:05.208711 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:05.208675    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"]
Sep 15 22:49:05.212412 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:05.212386    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fffdae-openshift-default-6f66bd6b74-956kw"]
Sep 15 22:49:05.492292 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:05.492182    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:05.898762 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:05.898724    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" path="/var/lib/kubelet/pods/5bda438d-af76-4c2d-8b4e-f39d13c857ea/volumes"
Sep 15 22:49:07.191824 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:07.191779    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:08.446130 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:08.446097    2584 scope.go:117] "RemoveContainer" containerID="801bb4343c78a4962c4cdc784a3ae0b95c45a026cfc83628ee63d27d9e3fb01d"
Sep 15 22:49:08.454981 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:08.454959    2584 scope.go:117] "RemoveContainer" containerID="e3090123db9a5b281034d42d29709beac09052a7fd003f6d0b6f16e276b03fa2"
Sep 15 22:49:08.462794 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:08.462772    2584 scope.go:117] "RemoveContainer" containerID="eaa516b835d6225eec07d6ef527df49fd70c735ba2de8a3ad3f18cb777dde924"
Sep 15 22:49:10.376495 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.376460    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:10.403367 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.403334    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"]
Sep 15 22:49:10.403619 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.403590    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5" podUID="b4d5cd52-9645-421a-a037-613918ea98ed" containerName="payload-processing" containerID="cri-o://c8e0c7f207b3c9fa20ff10ae22c740e62e7619be092f6d3675e18beec8097e33" gracePeriod=30
Sep 15 22:49:10.444481 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.444443    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:10.545902 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.545855    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:10.577482 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.577427    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr"]
Sep 15 22:49:10.580278 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.580237    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765e6d-j7nwr"]
Sep 15 22:49:10.603793 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.603726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:10.667141 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.667118    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"
Sep 15 22:49:10.674686 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.674656    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d5cd52-9645-421a-a037-613918ea98ed-plugins-config-volume\") pod \"b4d5cd52-9645-421a-a037-613918ea98ed\" (UID: \"b4d5cd52-9645-421a-a037-613918ea98ed\") "
Sep 15 22:49:10.674830 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.674702    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znsr9\" (UniqueName: \"kubernetes.io/projected/b4d5cd52-9645-421a-a037-613918ea98ed-kube-api-access-znsr9\") pod \"b4d5cd52-9645-421a-a037-613918ea98ed\" (UID: \"b4d5cd52-9645-421a-a037-613918ea98ed\") "
Sep 15 22:49:10.675104 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.675072    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d5cd52-9645-421a-a037-613918ea98ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4d5cd52-9645-421a-a037-613918ea98ed" (UID: "b4d5cd52-9645-421a-a037-613918ea98ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:49:10.676735 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.676708    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d5cd52-9645-421a-a037-613918ea98ed-kube-api-access-znsr9" (OuterVolumeSpecName: "kube-api-access-znsr9") pod "b4d5cd52-9645-421a-a037-613918ea98ed" (UID: "b4d5cd52-9645-421a-a037-613918ea98ed"). InnerVolumeSpecName "kube-api-access-znsr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:49:10.776020 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.775985    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d5cd52-9645-421a-a037-613918ea98ed-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:49:10.776020 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.776016    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znsr9\" (UniqueName: \"kubernetes.io/projected/b4d5cd52-9645-421a-a037-613918ea98ed-kube-api-access-znsr9\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:49:10.921093 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.921009    2584 generic.go:356] "Generic (PLEG): container finished" podID="b4d5cd52-9645-421a-a037-613918ea98ed" containerID="c8e0c7f207b3c9fa20ff10ae22c740e62e7619be092f6d3675e18beec8097e33" exitCode=0
Sep 15 22:49:10.921093 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.921060    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"
Sep 15 22:49:10.921344 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.921086    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5" event={"ID":"b4d5cd52-9645-421a-a037-613918ea98ed","Type":"ContainerDied","Data":"c8e0c7f207b3c9fa20ff10ae22c740e62e7619be092f6d3675e18beec8097e33"}
Sep 15 22:49:10.921344 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.921134    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5" event={"ID":"b4d5cd52-9645-421a-a037-613918ea98ed","Type":"ContainerDied","Data":"aa2c121f403193e7c6fe7782d1bcce0d9cb8a7eb91ccf20eaf0a828e994a1c9a"}
Sep 15 22:49:10.921344 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.921156    2584 scope.go:117] "RemoveContainer" containerID="c8e0c7f207b3c9fa20ff10ae22c740e62e7619be092f6d3675e18beec8097e33"
Sep 15 22:49:10.927316 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.927262    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:10.931488 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.931473    2584 scope.go:117] "RemoveContainer" containerID="c8e0c7f207b3c9fa20ff10ae22c740e62e7619be092f6d3675e18beec8097e33"
Sep 15 22:49:10.931776 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:49:10.931751    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e0c7f207b3c9fa20ff10ae22c740e62e7619be092f6d3675e18beec8097e33\": container with ID starting with c8e0c7f207b3c9fa20ff10ae22c740e62e7619be092f6d3675e18beec8097e33 not found: ID does not exist" containerID="c8e0c7f207b3c9fa20ff10ae22c740e62e7619be092f6d3675e18beec8097e33"
Sep 15 22:49:10.931848 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.931788    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e0c7f207b3c9fa20ff10ae22c740e62e7619be092f6d3675e18beec8097e33"} err="failed to get container status \"c8e0c7f207b3c9fa20ff10ae22c740e62e7619be092f6d3675e18beec8097e33\": rpc error: code = NotFound desc = could not find container \"c8e0c7f207b3c9fa20ff10ae22c740e62e7619be092f6d3675e18beec8097e33\": container with ID starting with c8e0c7f207b3c9fa20ff10ae22c740e62e7619be092f6d3675e18beec8097e33 not found: ID does not exist"
Sep 15 22:49:10.943397 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.943374    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"]
Sep 15 22:49:10.946658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:10.946635    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765e6d-74f87df745-mxhr5"]
Sep 15 22:49:11.058596 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:11.058568    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:11.864547 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:11.864501    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:11.900256 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:11.900223    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a30fd544-0cf7-4af4-a729-37e2a872b4b5" path="/var/lib/kubelet/pods/a30fd544-0cf7-4af4-a729-37e2a872b4b5/volumes"
Sep 15 22:49:11.900649 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:11.900633    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4d5cd52-9645-421a-a037-613918ea98ed" path="/var/lib/kubelet/pods/b4d5cd52-9645-421a-a037-613918ea98ed/volumes"
Sep 15 22:49:12.634530 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:12.634494    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:13.340247 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:13.340210    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:14.172885 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:14.172849    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:14.914339 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:14.914305    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:15.380641 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:15.380606    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:16.752982 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:16.752941    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:16.815101 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:16.815071    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:17.003160 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:17.003074    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:17.555629 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:17.555577    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:17.855990 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:17.855895    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:17.943216 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:17.943181    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:18.977551 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:18.977516    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:49:20.141607 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:20.141569    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:20.956528 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:20.956492    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:21.047014 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:21.046980    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:21.850433 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:21.850399    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:22.700984 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:22.700943    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:23.407778 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:23.407747    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:24.255361 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:24.255331    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:24.957065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:24.957032    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:25.013635 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.013600    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:25.115473 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.115432    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:25.420095 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.420063    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:25.520963 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.520927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:25.530522 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.530496    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"]
Sep 15 22:49:25.530952 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.530925    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a30fd544-0cf7-4af4-a729-37e2a872b4b5" containerName="revoke-keys"
Sep 15 22:49:25.530952 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.530943    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a30fd544-0cf7-4af4-a729-37e2a872b4b5" containerName="revoke-keys"
Sep 15 22:49:25.531131 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.530958    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerName="istio-proxy"
Sep 15 22:49:25.531131 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.530967    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerName="istio-proxy"
Sep 15 22:49:25.531131 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.530985    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4d5cd52-9645-421a-a037-613918ea98ed" containerName="payload-processing"
Sep 15 22:49:25.531131 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.530993    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4d5cd52-9645-421a-a037-613918ea98ed" containerName="payload-processing"
Sep 15 22:49:25.531131 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.531129    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerName="istio-proxy"
Sep 15 22:49:25.531417 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.531142    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerName="istio-proxy"
Sep 15 22:49:25.531417 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.531152    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4d5cd52-9645-421a-a037-613918ea98ed" containerName="payload-processing"
Sep 15 22:49:25.531417 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.531163    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a30fd544-0cf7-4af4-a729-37e2a872b4b5" containerName="revoke-keys"
Sep 15 22:49:25.535952 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.535933    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"
Sep 15 22:49:25.538877 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.538856    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-0016d4\""
Sep 15 22:49:25.538975 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.538954    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-0016d4-dockercfg-rdv5s\""
Sep 15 22:49:25.544153 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.544130    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"]
Sep 15 22:49:25.567226 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.567205    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"]
Sep 15 22:49:25.567633 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.567619    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerName="istio-proxy"
Sep 15 22:49:25.567677 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.567634    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bda438d-af76-4c2d-8b4e-f39d13c857ea" containerName="istio-proxy"
Sep 15 22:49:25.570754 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.570739    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"
Sep 15 22:49:25.583863 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.583842    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"]
Sep 15 22:49:25.628250 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.628216    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rxm6\" (UniqueName: \"kubernetes.io/projected/f7c420a4-2764-4622-b52f-e48999b2fbb6-kube-api-access-5rxm6\") pod \"payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk\" (UID: \"f7c420a4-2764-4622-b52f-e48999b2fbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"
Sep 15 22:49:25.628250 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.628250    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wd77\" (UniqueName: \"kubernetes.io/projected/82d084a5-4f09-4188-8d16-57db849fb124-kube-api-access-6wd77\") pod \"payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf\" (UID: \"82d084a5-4f09-4188-8d16-57db849fb124\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"
Sep 15 22:49:25.628439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.628320    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c420a4-2764-4622-b52f-e48999b2fbb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk\" (UID: \"f7c420a4-2764-4622-b52f-e48999b2fbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"
Sep 15 22:49:25.628439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.628367    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d084a5-4f09-4188-8d16-57db849fb124-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf\" (UID: \"82d084a5-4f09-4188-8d16-57db849fb124\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"
Sep 15 22:49:25.729432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.729346    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rxm6\" (UniqueName: \"kubernetes.io/projected/f7c420a4-2764-4622-b52f-e48999b2fbb6-kube-api-access-5rxm6\") pod \"payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk\" (UID: \"f7c420a4-2764-4622-b52f-e48999b2fbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"
Sep 15 22:49:25.729432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.729381    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wd77\" (UniqueName: \"kubernetes.io/projected/82d084a5-4f09-4188-8d16-57db849fb124-kube-api-access-6wd77\") pod \"payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf\" (UID: \"82d084a5-4f09-4188-8d16-57db849fb124\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"
Sep 15 22:49:25.729432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.729417    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c420a4-2764-4622-b52f-e48999b2fbb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk\" (UID: \"f7c420a4-2764-4622-b52f-e48999b2fbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"
Sep 15 22:49:25.729698 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.729467    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d084a5-4f09-4188-8d16-57db849fb124-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf\" (UID: \"82d084a5-4f09-4188-8d16-57db849fb124\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"
Sep 15 22:49:25.730137 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.730118    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c420a4-2764-4622-b52f-e48999b2fbb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk\" (UID: \"f7c420a4-2764-4622-b52f-e48999b2fbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"
Sep 15 22:49:25.730183 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.730165    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d084a5-4f09-4188-8d16-57db849fb124-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf\" (UID: \"82d084a5-4f09-4188-8d16-57db849fb124\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"
Sep 15 22:49:25.738025 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.738002    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wd77\" (UniqueName: \"kubernetes.io/projected/82d084a5-4f09-4188-8d16-57db849fb124-kube-api-access-6wd77\") pod \"payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf\" (UID: \"82d084a5-4f09-4188-8d16-57db849fb124\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"
Sep 15 22:49:25.738155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.738010    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rxm6\" (UniqueName: \"kubernetes.io/projected/f7c420a4-2764-4622-b52f-e48999b2fbb6-kube-api-access-5rxm6\") pod \"payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk\" (UID: \"f7c420a4-2764-4622-b52f-e48999b2fbb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"
Sep 15 22:49:25.848228 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.848196    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"
Sep 15 22:49:25.879894 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:25.879861    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"
Sep 15 22:49:26.127493 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:26.127463    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"]
Sep 15 22:49:26.128275 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:49:26.128242    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c420a4_2764_4622_b52f_e48999b2fbb6.slice/crio-ef59b0194d0ea3937f63fb478b02ce20aa4e3b375f866bfbbd070bce5d55e9f8 WatchSource:0}: Error finding container ef59b0194d0ea3937f63fb478b02ce20aa4e3b375f866bfbbd070bce5d55e9f8: Status 404 returned error can't find the container with id ef59b0194d0ea3937f63fb478b02ce20aa4e3b375f866bfbbd070bce5d55e9f8
Sep 15 22:49:26.163720 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:26.163655    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"]
Sep 15 22:49:26.165215 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:49:26.165193    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d084a5_4f09_4188_8d16_57db849fb124.slice/crio-6abb77987d0b259f64cf5f3e150f99651f7a38c6cf1f8afee67b1ab34f72a2d4 WatchSource:0}: Error finding container 6abb77987d0b259f64cf5f3e150f99651f7a38c6cf1f8afee67b1ab34f72a2d4: Status 404 returned error can't find the container with id 6abb77987d0b259f64cf5f3e150f99651f7a38c6cf1f8afee67b1ab34f72a2d4
Sep 15 22:49:27.020311 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:27.020239    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf" event={"ID":"82d084a5-4f09-4188-8d16-57db849fb124","Type":"ContainerStarted","Data":"75972eb6c310ad7483078c50a3809f75d37f30bb63aa2107cfe9b9aff8a61e23"}
Sep 15 22:49:27.020509 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:27.020329    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf" event={"ID":"82d084a5-4f09-4188-8d16-57db849fb124","Type":"ContainerStarted","Data":"6abb77987d0b259f64cf5f3e150f99651f7a38c6cf1f8afee67b1ab34f72a2d4"}
Sep 15 22:49:27.020509 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:27.020445    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"
Sep 15 22:49:27.022124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:27.022095    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk" event={"ID":"f7c420a4-2764-4622-b52f-e48999b2fbb6","Type":"ContainerStarted","Data":"03508bfd029a3b8f70b09ec20977c3bbf496728fff9f8e04f6bf733209c49fd3"}
Sep 15 22:49:27.022240 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:27.022146    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk" event={"ID":"f7c420a4-2764-4622-b52f-e48999b2fbb6","Type":"ContainerStarted","Data":"ef59b0194d0ea3937f63fb478b02ce20aa4e3b375f866bfbbd070bce5d55e9f8"}
Sep 15 22:49:27.022240 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:27.022184    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"
Sep 15 22:49:27.039761 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:27.039718    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf" podStartSLOduration=2.03970332 podStartE2EDuration="2.03970332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:49:27.037413555 +0000 UTC m=+1339.656228463" watchObservedRunningTime="2026-09-15 22:49:27.03970332 +0000 UTC m=+1339.658518228"
Sep 15 22:49:27.053013 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:27.052967    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk" podStartSLOduration=2.05294922 podStartE2EDuration="2.05294922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:49:27.052856197 +0000 UTC m=+1339.671671106" watchObservedRunningTime="2026-09-15 22:49:27.05294922 +0000 UTC m=+1339.671764129"
Sep 15 22:49:28.340885 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.340846    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"]
Sep 15 22:49:28.344685 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.344663    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.347321 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.347250    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fe020d\"/\"distinct-2-w0-fe020d-kserve-self-signed-certs\""
Sep 15 22:49:28.351999 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.351966    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"]
Sep 15 22:49:28.464689 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.464659    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-model-cache\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.464817 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.464697    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-dshm\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.464817 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.464729    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-home\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.464817 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.464782    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bps2h\" (UniqueName: \"kubernetes.io/projected/8dafd07c-2342-4799-803a-061c6fe6488d-kube-api-access-bps2h\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.464817 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.464814    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-tmp-dir\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.464961 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.464856    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dafd07c-2342-4799-803a-061c6fe6488d-tls-certs\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.565892 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.565856    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dafd07c-2342-4799-803a-061c6fe6488d-tls-certs\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.565992 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.565911    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-model-cache\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.565992 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.565942    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-dshm\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.565992 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.565971    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-home\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.566131 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.566003    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bps2h\" (UniqueName: \"kubernetes.io/projected/8dafd07c-2342-4799-803a-061c6fe6488d-kube-api-access-bps2h\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.566131 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.566085    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-tmp-dir\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.566460 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.566440    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-model-cache\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.566570 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.566480    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-home\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.566570 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.566500    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-tmp-dir\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.568232 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.568208    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-dshm\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.568446 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.568429    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dafd07c-2342-4799-803a-061c6fe6488d-tls-certs\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.575098 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.575077    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bps2h\" (UniqueName: \"kubernetes.io/projected/8dafd07c-2342-4799-803a-061c6fe6488d-kube-api-access-bps2h\") pod \"distinct-2-w0-fe020d-kserve-f89687c95-vmnkh\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") " pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.650649 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.650583    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:28.655132 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.655111    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:28.928719 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:28.928692    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"]
Sep 15 22:49:28.930860 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:49:28.930834    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dafd07c_2342_4799_803a_061c6fe6488d.slice/crio-9ff2c79bd35a831c0d68cf19e3f43672fd1d51c84f12b5ed3244469939a38945 WatchSource:0}: Error finding container 9ff2c79bd35a831c0d68cf19e3f43672fd1d51c84f12b5ed3244469939a38945: Status 404 returned error can't find the container with id 9ff2c79bd35a831c0d68cf19e3f43672fd1d51c84f12b5ed3244469939a38945
Sep 15 22:49:29.035102 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:29.035064    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh" event={"ID":"8dafd07c-2342-4799-803a-061c6fe6488d","Type":"ContainerStarted","Data":"00992e64db88b3c6091bfa3a46a7aeae491979c38fb8c75aed3fa3e0fdd9ede1"}
Sep 15 22:49:29.035291 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:29.035118    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh" event={"ID":"8dafd07c-2342-4799-803a-061c6fe6488d","Type":"ContainerStarted","Data":"9ff2c79bd35a831c0d68cf19e3f43672fd1d51c84f12b5ed3244469939a38945"}
Sep 15 22:49:29.053970 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:29.053919    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh" podStartSLOduration=1.053897805 podStartE2EDuration="1.053897805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:49:29.052498258 +0000 UTC m=+1341.671313167" watchObservedRunningTime="2026-09-15 22:49:29.053897805 +0000 UTC m=+1341.672712714"
Sep 15 22:49:38.030412 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:38.030323    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"
Sep 15 22:49:38.031030 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:38.030441    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"
Sep 15 22:49:38.655441 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:38.655409    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:38.655617 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:38.655467    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:38.667846 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:38.667819    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:39.105445 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:39.105415    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:49:45.572935 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:45.572901    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf"]
Sep 15 22:49:45.576615 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:45.576594    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf"
Sep 15 22:49:45.579286 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:45.579236    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:49:45.579401 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:45.579299    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:49:45.584666 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:45.584545    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf"]
Sep 15 22:49:45.629417 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:45.629391    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fa66f9-22e7-472f-9bf5-d97924553f95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf\" (UID: \"f9fa66f9-22e7-472f-9bf5-d97924553f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf"
Sep 15 22:49:45.730237 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:45.730194    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fa66f9-22e7-472f-9bf5-d97924553f95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf\" (UID: \"f9fa66f9-22e7-472f-9bf5-d97924553f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf"
Sep 15 22:49:45.730837 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:45.730819    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fa66f9-22e7-472f-9bf5-d97924553f95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf\" (UID: \"f9fa66f9-22e7-472f-9bf5-d97924553f95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf"
Sep 15 22:49:45.881335 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:45.881200    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:45.894024 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:45.894003    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf"
Sep 15 22:49:46.168139 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:46.168113    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf"]
Sep 15 22:49:46.170237 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:49:46.170210    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fa66f9_22e7_472f_9bf5_d97924553f95.slice/crio-50654d829749ea5dd5fbb9209771acbd36d0c62862da5354fdb5fc3e3609ece9 WatchSource:0}: Error finding container 50654d829749ea5dd5fbb9209771acbd36d0c62862da5354fdb5fc3e3609ece9: Status 404 returned error can't find the container with id 50654d829749ea5dd5fbb9209771acbd36d0c62862da5354fdb5fc3e3609ece9
Sep 15 22:49:46.680221 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:46.680187    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:47.150135 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:47.150046    2584 generic.go:356] "Generic (PLEG): container finished" podID="f9fa66f9-22e7-472f-9bf5-d97924553f95" containerID="5e34cd033b2f10e39b8b605bdaccc3c805c20baf7728e230340b30883b3cd95c" exitCode=0
Sep 15 22:49:47.150135 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:47.150116    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf" event={"ID":"f9fa66f9-22e7-472f-9bf5-d97924553f95","Type":"ContainerDied","Data":"5e34cd033b2f10e39b8b605bdaccc3c805c20baf7728e230340b30883b3cd95c"}
Sep 15 22:49:47.150346 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:47.150176    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf" event={"ID":"f9fa66f9-22e7-472f-9bf5-d97924553f95","Type":"ContainerStarted","Data":"50654d829749ea5dd5fbb9209771acbd36d0c62862da5354fdb5fc3e3609ece9"}
Sep 15 22:49:47.436829 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:47.436732    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:48.154186 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:48.154150    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:48.292067 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:48.292045    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf"
Sep 15 22:49:48.354177 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:48.354148    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fa66f9-22e7-472f-9bf5-d97924553f95-maas-api-service-ca\") pod \"f9fa66f9-22e7-472f-9bf5-d97924553f95\" (UID: \"f9fa66f9-22e7-472f-9bf5-d97924553f95\") "
Sep 15 22:49:48.354528 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:48.354505    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fa66f9-22e7-472f-9bf5-d97924553f95-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f9fa66f9-22e7-472f-9bf5-d97924553f95" (UID: "f9fa66f9-22e7-472f-9bf5-d97924553f95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:49:48.455299 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:48.455202    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fa66f9-22e7-472f-9bf5-d97924553f95-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:49:49.066210 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:49.066176    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:49.164015 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:49.163980    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf" event={"ID":"f9fa66f9-22e7-472f-9bf5-d97924553f95","Type":"ContainerDied","Data":"50654d829749ea5dd5fbb9209771acbd36d0c62862da5354fdb5fc3e3609ece9"}
Sep 15 22:49:49.164432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:49.164028    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50654d829749ea5dd5fbb9209771acbd36d0c62862da5354fdb5fc3e3609ece9"
Sep 15 22:49:49.164432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:49.163989    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf"
Sep 15 22:49:49.784989 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:49.784951    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:49.860098 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:49.860005    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:50.643594 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:50.643556    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:50.907082 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:50.907000    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:51.172178 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:51.172092    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"]
Sep 15 22:49:51.192156 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:51.192121    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:51.288849 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:51.288813    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:51.464619 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:51.464533    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:51.551709 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:51.551672    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:51.957619 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:51.957581    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:52.246068 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:52.245975    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:53.066181 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:53.066148    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:54.280345 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:54.280304    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:55.164491 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:55.164454    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:55.262915 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:55.262878    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:56.050591 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:56.050558    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:57.069038 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:57.069005    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:57.747959 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:57.747923    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:57.961074 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:57.961041    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:49:58.519337 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:58.519294    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:58.825493 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:58.825461    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:59.299800 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.299764    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:59.343194 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.343153    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"]
Sep 15 22:49:59.343693 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.343677    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fa66f9-22e7-472f-9bf5-d97924553f95" containerName="revoke-keys"
Sep 15 22:49:59.343693 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.343694    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fa66f9-22e7-472f-9bf5-d97924553f95" containerName="revoke-keys"
Sep 15 22:49:59.343797 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.343770    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fa66f9-22e7-472f-9bf5-d97924553f95" containerName="revoke-keys"
Sep 15 22:49:59.348289 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.348257    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.350852 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.350831    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-03f258\"/\"distinct-w4-03f258-kserve-self-signed-certs\""
Sep 15 22:49:59.355419 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.355398    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"]
Sep 15 22:49:59.400134 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.400098    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:59.468140 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.468112    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-dshm\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.468319 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.468150    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-tmp-dir\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.468319 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.468177    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-home\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.468319 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.468207    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-model-cache\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.468319 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.468226    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krpdt\" (UniqueName: \"kubernetes.io/projected/3829049a-52f5-44b9-8ea2-de5558590665-kube-api-access-krpdt\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.468319 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.468245    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3829049a-52f5-44b9-8ea2-de5558590665-tls-certs\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.529819 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.529790    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:59.569758 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.569698    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-dshm\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.569758 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.569732    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-tmp-dir\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.569924 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.569762    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-home\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.569924 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.569785    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-model-cache\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.569924 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.569812    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krpdt\" (UniqueName: \"kubernetes.io/projected/3829049a-52f5-44b9-8ea2-de5558590665-kube-api-access-krpdt\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.569924 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.569834    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3829049a-52f5-44b9-8ea2-de5558590665-tls-certs\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.570209 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.570183    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-tmp-dir\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.570327 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.570241    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-model-cache\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.570327 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.570288    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-home\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.571934 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.571912    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-dshm\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.572190 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.572173    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3829049a-52f5-44b9-8ea2-de5558590665-tls-certs\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.577792 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.577775    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krpdt\" (UniqueName: \"kubernetes.io/projected/3829049a-52f5-44b9-8ea2-de5558590665-kube-api-access-krpdt\") pod \"distinct-w4-03f258-kserve-76585b58ff-brfg6\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.660488 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.660442    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:49:59.907365 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.907261    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:49:59.937436 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:49:59.937406    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"]
Sep 15 22:49:59.938699 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:49:59.938668    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3829049a_52f5_44b9_8ea2_de5558590665.slice/crio-e01adf8cd1695eb2efe5cd75a4d339ee586dda1dc89d12e744841d5b5358727b WatchSource:0}: Error finding container e01adf8cd1695eb2efe5cd75a4d339ee586dda1dc89d12e744841d5b5358727b: Status 404 returned error can't find the container with id e01adf8cd1695eb2efe5cd75a4d339ee586dda1dc89d12e744841d5b5358727b
Sep 15 22:50:00.004971 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:00.004941    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:00.234969 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:00.234885    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6" event={"ID":"3829049a-52f5-44b9-8ea2-de5558590665","Type":"ContainerStarted","Data":"827a28b966e9235cf4f9d330c46fbb943aa1051a26a073e1b79d5349a5405b1f"}
Sep 15 22:50:00.234969 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:00.234930    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6" event={"ID":"3829049a-52f5-44b9-8ea2-de5558590665","Type":"ContainerStarted","Data":"e01adf8cd1695eb2efe5cd75a4d339ee586dda1dc89d12e744841d5b5358727b"}
Sep 15 22:50:00.253279 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:00.253225    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6" podStartSLOduration=1.253210762 podStartE2EDuration="1.253210762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:50:00.250784732 +0000 UTC m=+1372.869599643" watchObservedRunningTime="2026-09-15 22:50:00.253210762 +0000 UTC m=+1372.872025669"
Sep 15 22:50:00.327339 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:00.327311    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:01.176849 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:01.176813    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:01.780386 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:01.777811    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:01.808882 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:01.808853    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"]
Sep 15 22:50:01.809087 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:01.809060    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv" podUID="ea649cda-4214-472b-8494-c9d851de1cc3" containerName="payload-processing" containerID="cri-o://c0f6be97b50dc79345222175d5a0459b58e2ba8909d04b89b97704693165d2a1" gracePeriod=30
Sep 15 22:50:01.817159 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:01.816766    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"]
Sep 15 22:50:01.817159 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:01.816976    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq" podUID="c94e239b-0a13-4d3a-b497-2dbd34631404" containerName="payload-pre-processing" containerID="cri-o://9ffa0b826975f04934059cd9d34ebaba6331f07bfb1bd9e9210a1324c7b3abe7" gracePeriod=30
Sep 15 22:50:01.853326 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:01.853296    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:01.916695 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:01.916663    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:01.993202 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:01.993167    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf"]
Sep 15 22:50:01.995762 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:01.995731    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6cdcda22-bzcvf"]
Sep 15 22:50:02.038614 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.038531    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:02.114872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.114836    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:02.115414 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.115388    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"
Sep 15 22:50:02.139093 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.139061    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"
Sep 15 22:50:02.199386 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.199355    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ffz7\" (UniqueName: \"kubernetes.io/projected/c94e239b-0a13-4d3a-b497-2dbd34631404-kube-api-access-6ffz7\") pod \"c94e239b-0a13-4d3a-b497-2dbd34631404\" (UID: \"c94e239b-0a13-4d3a-b497-2dbd34631404\") "
Sep 15 22:50:02.199879 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.199399    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94e239b-0a13-4d3a-b497-2dbd34631404-plugins-config-volume\") pod \"c94e239b-0a13-4d3a-b497-2dbd34631404\" (UID: \"c94e239b-0a13-4d3a-b497-2dbd34631404\") "
Sep 15 22:50:02.199879 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.199426    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6t8f\" (UniqueName: \"kubernetes.io/projected/ea649cda-4214-472b-8494-c9d851de1cc3-kube-api-access-q6t8f\") pod \"ea649cda-4214-472b-8494-c9d851de1cc3\" (UID: \"ea649cda-4214-472b-8494-c9d851de1cc3\") "
Sep 15 22:50:02.199879 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.199472    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea649cda-4214-472b-8494-c9d851de1cc3-plugins-config-volume\") pod \"ea649cda-4214-472b-8494-c9d851de1cc3\" (UID: \"ea649cda-4214-472b-8494-c9d851de1cc3\") "
Sep 15 22:50:02.200055 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.199894    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94e239b-0a13-4d3a-b497-2dbd34631404-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c94e239b-0a13-4d3a-b497-2dbd34631404" (UID: "c94e239b-0a13-4d3a-b497-2dbd34631404"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:50:02.200055 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.199944    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea649cda-4214-472b-8494-c9d851de1cc3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea649cda-4214-472b-8494-c9d851de1cc3" (UID: "ea649cda-4214-472b-8494-c9d851de1cc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:50:02.201545 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.201521    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea649cda-4214-472b-8494-c9d851de1cc3-kube-api-access-q6t8f" (OuterVolumeSpecName: "kube-api-access-q6t8f") pod "ea649cda-4214-472b-8494-c9d851de1cc3" (UID: "ea649cda-4214-472b-8494-c9d851de1cc3"). InnerVolumeSpecName "kube-api-access-q6t8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:50:02.202072 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.202046    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94e239b-0a13-4d3a-b497-2dbd34631404-kube-api-access-6ffz7" (OuterVolumeSpecName: "kube-api-access-6ffz7") pod "c94e239b-0a13-4d3a-b497-2dbd34631404" (UID: "c94e239b-0a13-4d3a-b497-2dbd34631404"). InnerVolumeSpecName "kube-api-access-6ffz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:50:02.248122 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.248088    2584 generic.go:356] "Generic (PLEG): container finished" podID="c94e239b-0a13-4d3a-b497-2dbd34631404" containerID="9ffa0b826975f04934059cd9d34ebaba6331f07bfb1bd9e9210a1324c7b3abe7" exitCode=0
Sep 15 22:50:02.248326 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.248133    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"
Sep 15 22:50:02.248326 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.248184    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq" event={"ID":"c94e239b-0a13-4d3a-b497-2dbd34631404","Type":"ContainerDied","Data":"9ffa0b826975f04934059cd9d34ebaba6331f07bfb1bd9e9210a1324c7b3abe7"}
Sep 15 22:50:02.248326 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.248245    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq" event={"ID":"c94e239b-0a13-4d3a-b497-2dbd34631404","Type":"ContainerDied","Data":"9642288b3ab225432944c36373273cb618d12aa184285c75bcc275cf2d857f2e"}
Sep 15 22:50:02.248326 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.248308    2584 scope.go:117] "RemoveContainer" containerID="9ffa0b826975f04934059cd9d34ebaba6331f07bfb1bd9e9210a1324c7b3abe7"
Sep 15 22:50:02.249717 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.249694    2584 generic.go:356] "Generic (PLEG): container finished" podID="ea649cda-4214-472b-8494-c9d851de1cc3" containerID="c0f6be97b50dc79345222175d5a0459b58e2ba8909d04b89b97704693165d2a1" exitCode=0
Sep 15 22:50:02.249871 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.249740    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"
Sep 15 22:50:02.249871 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.249778    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv" event={"ID":"ea649cda-4214-472b-8494-c9d851de1cc3","Type":"ContainerDied","Data":"c0f6be97b50dc79345222175d5a0459b58e2ba8909d04b89b97704693165d2a1"}
Sep 15 22:50:02.249871 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.249818    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv" event={"ID":"ea649cda-4214-472b-8494-c9d851de1cc3","Type":"ContainerDied","Data":"8c5f2fdcdacc6af71bdb44c9d557e0142ca443e70fdb24669a219e3d3a06bae7"}
Sep 15 22:50:02.261049 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.261033    2584 scope.go:117] "RemoveContainer" containerID="9ffa0b826975f04934059cd9d34ebaba6331f07bfb1bd9e9210a1324c7b3abe7"
Sep 15 22:50:02.261363 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:50:02.261319    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ffa0b826975f04934059cd9d34ebaba6331f07bfb1bd9e9210a1324c7b3abe7\": container with ID starting with 9ffa0b826975f04934059cd9d34ebaba6331f07bfb1bd9e9210a1324c7b3abe7 not found: ID does not exist" containerID="9ffa0b826975f04934059cd9d34ebaba6331f07bfb1bd9e9210a1324c7b3abe7"
Sep 15 22:50:02.261441 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.261354    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ffa0b826975f04934059cd9d34ebaba6331f07bfb1bd9e9210a1324c7b3abe7"} err="failed to get container status \"9ffa0b826975f04934059cd9d34ebaba6331f07bfb1bd9e9210a1324c7b3abe7\": rpc error: code = NotFound desc = could not find container \"9ffa0b826975f04934059cd9d34ebaba6331f07bfb1bd9e9210a1324c7b3abe7\": container with ID starting with 9ffa0b826975f04934059cd9d34ebaba6331f07bfb1bd9e9210a1324c7b3abe7 not found: ID does not exist"
Sep 15 22:50:02.261441 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.261379    2584 scope.go:117] "RemoveContainer" containerID="c0f6be97b50dc79345222175d5a0459b58e2ba8909d04b89b97704693165d2a1"
Sep 15 22:50:02.270150 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.270129    2584 scope.go:117] "RemoveContainer" containerID="c0f6be97b50dc79345222175d5a0459b58e2ba8909d04b89b97704693165d2a1"
Sep 15 22:50:02.270423 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:50:02.270404    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f6be97b50dc79345222175d5a0459b58e2ba8909d04b89b97704693165d2a1\": container with ID starting with c0f6be97b50dc79345222175d5a0459b58e2ba8909d04b89b97704693165d2a1 not found: ID does not exist" containerID="c0f6be97b50dc79345222175d5a0459b58e2ba8909d04b89b97704693165d2a1"
Sep 15 22:50:02.270481 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.270430    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f6be97b50dc79345222175d5a0459b58e2ba8909d04b89b97704693165d2a1"} err="failed to get container status \"c0f6be97b50dc79345222175d5a0459b58e2ba8909d04b89b97704693165d2a1\": rpc error: code = NotFound desc = could not find container \"c0f6be97b50dc79345222175d5a0459b58e2ba8909d04b89b97704693165d2a1\": container with ID starting with c0f6be97b50dc79345222175d5a0459b58e2ba8909d04b89b97704693165d2a1 not found: ID does not exist"
Sep 15 22:50:02.277941 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.277915    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"]
Sep 15 22:50:02.284159 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.284129    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6cdcda22-586bcff56b-s6vhq"]
Sep 15 22:50:02.300869 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.300841    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea649cda-4214-472b-8494-c9d851de1cc3-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:50:02.300869 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.300869    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ffz7\" (UniqueName: \"kubernetes.io/projected/c94e239b-0a13-4d3a-b497-2dbd34631404-kube-api-access-6ffz7\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:50:02.301061 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.300881    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94e239b-0a13-4d3a-b497-2dbd34631404-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:50:02.301061 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.300890    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6t8f\" (UniqueName: \"kubernetes.io/projected/ea649cda-4214-472b-8494-c9d851de1cc3-kube-api-access-q6t8f\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:50:02.309465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.309441    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"]
Sep 15 22:50:02.312222 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.312196    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6cdcda22-5c64d9ff77-bbssv"]
Sep 15 22:50:02.470089 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:02.470053    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:03.191699 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:03.191579    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:03.900524 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:03.900488    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:03.900910 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:03.900676    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c94e239b-0a13-4d3a-b497-2dbd34631404" path="/var/lib/kubelet/pods/c94e239b-0a13-4d3a-b497-2dbd34631404/volumes"
Sep 15 22:50:03.901148 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:03.901130    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea649cda-4214-472b-8494-c9d851de1cc3" path="/var/lib/kubelet/pods/ea649cda-4214-472b-8494-c9d851de1cc3/volumes"
Sep 15 22:50:03.901500 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:03.901485    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9fa66f9-22e7-472f-9bf5-d97924553f95" path="/var/lib/kubelet/pods/f9fa66f9-22e7-472f-9bf5-d97924553f95/volumes"
Sep 15 22:50:04.672791 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:04.672756    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:05.096225 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:05.096183    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:05.384379 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:05.384298    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:06.192182 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.192136    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb" podUID="65d92989-df52-4f5d-b2d0-899af4dcc946" containerName="main" containerID="cri-o://600ca232ace596ad4d94a28a6fb23b93c073e4c6d8c13dda0e121e4d5b2d5299" gracePeriod=45
Sep 15 22:50:06.249376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.249341    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:06.451353 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.451289    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:50:06.554112 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.554083    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-tmp-dir\") pod \"65d92989-df52-4f5d-b2d0-899af4dcc946\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") "
Sep 15 22:50:06.554320 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.554123    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d92989-df52-4f5d-b2d0-899af4dcc946-tls-certs\") pod \"65d92989-df52-4f5d-b2d0-899af4dcc946\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") "
Sep 15 22:50:06.554320 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.554148    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-model-cache\") pod \"65d92989-df52-4f5d-b2d0-899af4dcc946\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") "
Sep 15 22:50:06.554320 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.554169    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-dshm\") pod \"65d92989-df52-4f5d-b2d0-899af4dcc946\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") "
Sep 15 22:50:06.554320 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.554222    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-874bd\" (UniqueName: \"kubernetes.io/projected/65d92989-df52-4f5d-b2d0-899af4dcc946-kube-api-access-874bd\") pod \"65d92989-df52-4f5d-b2d0-899af4dcc946\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") "
Sep 15 22:50:06.554320 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.554251    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-home\") pod \"65d92989-df52-4f5d-b2d0-899af4dcc946\" (UID: \"65d92989-df52-4f5d-b2d0-899af4dcc946\") "
Sep 15 22:50:06.554605 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.554414    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65d92989-df52-4f5d-b2d0-899af4dcc946" (UID: "65d92989-df52-4f5d-b2d0-899af4dcc946"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:50:06.554605 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.554440    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-model-cache" (OuterVolumeSpecName: "model-cache") pod "65d92989-df52-4f5d-b2d0-899af4dcc946" (UID: "65d92989-df52-4f5d-b2d0-899af4dcc946"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:50:06.554605 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.554584    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-home" (OuterVolumeSpecName: "home") pod "65d92989-df52-4f5d-b2d0-899af4dcc946" (UID: "65d92989-df52-4f5d-b2d0-899af4dcc946"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:50:06.554766 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.554734    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:50:06.554766 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.554762    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:50:06.554874 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.554776    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:50:06.556590 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.556562    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d92989-df52-4f5d-b2d0-899af4dcc946-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65d92989-df52-4f5d-b2d0-899af4dcc946" (UID: "65d92989-df52-4f5d-b2d0-899af4dcc946"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:50:06.556707 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.556607    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-dshm" (OuterVolumeSpecName: "dshm") pod "65d92989-df52-4f5d-b2d0-899af4dcc946" (UID: "65d92989-df52-4f5d-b2d0-899af4dcc946"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:50:06.556707 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.556614    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d92989-df52-4f5d-b2d0-899af4dcc946-kube-api-access-874bd" (OuterVolumeSpecName: "kube-api-access-874bd") pod "65d92989-df52-4f5d-b2d0-899af4dcc946" (UID: "65d92989-df52-4f5d-b2d0-899af4dcc946"). InnerVolumeSpecName "kube-api-access-874bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:50:06.655883 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.655854    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d92989-df52-4f5d-b2d0-899af4dcc946-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:50:06.655883 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.655877    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d92989-df52-4f5d-b2d0-899af4dcc946-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:50:06.655883 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.655887    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-874bd\" (UniqueName: \"kubernetes.io/projected/65d92989-df52-4f5d-b2d0-899af4dcc946-kube-api-access-874bd\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:50:06.954317 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:06.954279    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:07.285792 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:07.285694    2584 generic.go:356] "Generic (PLEG): container finished" podID="65d92989-df52-4f5d-b2d0-899af4dcc946" containerID="600ca232ace596ad4d94a28a6fb23b93c073e4c6d8c13dda0e121e4d5b2d5299" exitCode=0
Sep 15 22:50:07.285792 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:07.285738    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"
Sep 15 22:50:07.285792 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:07.285777    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb" event={"ID":"65d92989-df52-4f5d-b2d0-899af4dcc946","Type":"ContainerDied","Data":"600ca232ace596ad4d94a28a6fb23b93c073e4c6d8c13dda0e121e4d5b2d5299"}
Sep 15 22:50:07.286378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:07.285818    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb" event={"ID":"65d92989-df52-4f5d-b2d0-899af4dcc946","Type":"ContainerDied","Data":"c4c899dd1ee69b1da19a61586836f1d0b707d06f7b1703ace7a6123095c3751c"}
Sep 15 22:50:07.286378 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:07.285841    2584 scope.go:117] "RemoveContainer" containerID="600ca232ace596ad4d94a28a6fb23b93c073e4c6d8c13dda0e121e4d5b2d5299"
Sep 15 22:50:07.296381 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:07.296362    2584 scope.go:117] "RemoveContainer" containerID="600ca232ace596ad4d94a28a6fb23b93c073e4c6d8c13dda0e121e4d5b2d5299"
Sep 15 22:50:07.296658 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:50:07.296639    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600ca232ace596ad4d94a28a6fb23b93c073e4c6d8c13dda0e121e4d5b2d5299\": container with ID starting with 600ca232ace596ad4d94a28a6fb23b93c073e4c6d8c13dda0e121e4d5b2d5299 not found: ID does not exist" containerID="600ca232ace596ad4d94a28a6fb23b93c073e4c6d8c13dda0e121e4d5b2d5299"
Sep 15 22:50:07.296718 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:07.296671    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600ca232ace596ad4d94a28a6fb23b93c073e4c6d8c13dda0e121e4d5b2d5299"} err="failed to get container status \"600ca232ace596ad4d94a28a6fb23b93c073e4c6d8c13dda0e121e4d5b2d5299\": rpc error: code = NotFound desc = could not find container \"600ca232ace596ad4d94a28a6fb23b93c073e4c6d8c13dda0e121e4d5b2d5299\": container with ID starting with 600ca232ace596ad4d94a28a6fb23b93c073e4c6d8c13dda0e121e4d5b2d5299 not found: ID does not exist"
Sep 15 22:50:07.308973 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:07.308947    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"]
Sep 15 22:50:07.311256 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:07.311233    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/e2e-conflict-a-51df66af-kserve-77fddfbf8d-bz5vb"]
Sep 15 22:50:07.902150 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:07.902117    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65d92989-df52-4f5d-b2d0-899af4dcc946" path="/var/lib/kubelet/pods/65d92989-df52-4f5d-b2d0-899af4dcc946/volumes"
Sep 15 22:50:09.661366 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:09.661335    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:50:09.661833 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:09.661602    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:50:09.674365 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:09.674337    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:50:10.150113 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:10.150078    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:10.315685 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:10.315654    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:50:20.477470 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:20.477406    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:21.761258 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:21.761209    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:22.231978 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:22.231941    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:22.621959 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:22.621922    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:23.443224 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:23.443190    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:24.355415 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:24.355378    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:25.020500 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:25.020463    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:25.121511 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:25.121475    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:25.853817 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:25.853782    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:26.669150 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:26.669114    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:33.665241 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:33.665201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:35.260932 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:35.260896    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:50:43.830934 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:43.830897    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:46.556106 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:46.556070    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:47.281872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:47.281831    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:52.052518 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:52.052478    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:52.910597 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:52.910564    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:53.674298 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:53.674247    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:54.016352 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:54.016219    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:56.884419 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:56.884375    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:50:58.458486 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:50:58.458452    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:51:06.542475 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.542380    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"]
Sep 15 22:51:06.543043 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.543024    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d92989-df52-4f5d-b2d0-899af4dcc946" containerName="main"
Sep 15 22:51:06.543155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.543047    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d92989-df52-4f5d-b2d0-899af4dcc946" containerName="main"
Sep 15 22:51:06.543155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.543079    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c94e239b-0a13-4d3a-b497-2dbd34631404" containerName="payload-pre-processing"
Sep 15 22:51:06.543155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.543088    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c94e239b-0a13-4d3a-b497-2dbd34631404" containerName="payload-pre-processing"
Sep 15 22:51:06.543155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.543108    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea649cda-4214-472b-8494-c9d851de1cc3" containerName="payload-processing"
Sep 15 22:51:06.543155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.543119    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea649cda-4214-472b-8494-c9d851de1cc3" containerName="payload-processing"
Sep 15 22:51:06.543432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.543258    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c94e239b-0a13-4d3a-b497-2dbd34631404" containerName="payload-pre-processing"
Sep 15 22:51:06.543432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.543377    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea649cda-4214-472b-8494-c9d851de1cc3" containerName="payload-processing"
Sep 15 22:51:06.543432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.543396    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="65d92989-df52-4f5d-b2d0-899af4dcc946" containerName="main"
Sep 15 22:51:06.547157 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.547138    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.550648 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.550629    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-03f258\"/\"distinct-2-w4-03f258-kserve-self-signed-certs\""
Sep 15 22:51:06.555972 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.555951    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"]
Sep 15 22:51:06.565458 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.565426    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:06.632859 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.632829    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20834e61-d5e9-4fb3-8fc9-1939228873c0-tls-certs\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.632989 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.632925    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-tmp-dir\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.632989 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.632979    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvpjd\" (UniqueName: \"kubernetes.io/projected/20834e61-d5e9-4fb3-8fc9-1939228873c0-kube-api-access-gvpjd\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.633092 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.633025    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-dshm\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.633143 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.633113    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-home\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.633186 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.633149    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-model-cache\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.733992 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.733953    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvpjd\" (UniqueName: \"kubernetes.io/projected/20834e61-d5e9-4fb3-8fc9-1939228873c0-kube-api-access-gvpjd\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.734173 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.734076    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-dshm\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.734173 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.734133    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-home\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.734332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.734229    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-model-cache\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.734332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.734320    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20834e61-d5e9-4fb3-8fc9-1939228873c0-tls-certs\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.734451 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.734377    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-tmp-dir\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.734507 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.734456    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-home\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.734614 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.734596    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-model-cache\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.734709 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.734688    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-tmp-dir\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.736407 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.736378    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-dshm\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.736677 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.736659    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20834e61-d5e9-4fb3-8fc9-1939228873c0-tls-certs\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.741746 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.741727    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvpjd\" (UniqueName: \"kubernetes.io/projected/20834e61-d5e9-4fb3-8fc9-1939228873c0-kube-api-access-gvpjd\") pod \"distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") " pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:06.858802 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:06.858760    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:07.098723 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:07.098681    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:07.132175 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:07.132149    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"]
Sep 15 22:51:07.134288 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:51:07.134227    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20834e61_d5e9_4fb3_8fc9_1939228873c0.slice/crio-a7dcf51aefbf51ecb189ed5a42e1920f1342dab38058da24c297a9a6d798d7c3 WatchSource:0}: Error finding container a7dcf51aefbf51ecb189ed5a42e1920f1342dab38058da24c297a9a6d798d7c3: Status 404 returned error can't find the container with id a7dcf51aefbf51ecb189ed5a42e1920f1342dab38058da24c297a9a6d798d7c3
Sep 15 22:51:07.198968 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:07.198936    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:07.335923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:07.335885    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"]
Sep 15 22:51:07.336178 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:07.336135    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z" podUID="40d601f3-2361-4855-8090-643aa8582094" containerName="istio-proxy" containerID="cri-o://c69a9b8174db96c58ad145b6eee1864d60549d45753c682a2f367446fb45e648" gracePeriod=30
Sep 15 22:51:07.350424 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:07.350395    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:07.644138 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:07.644101    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4" event={"ID":"20834e61-d5e9-4fb3-8fc9-1939228873c0","Type":"ContainerStarted","Data":"6f01a878ac2a629168b6c81e602ea836ca7c475f6d75c43b931df30f5f4832fa"}
Sep 15 22:51:07.644138 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:07.644149    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4" event={"ID":"20834e61-d5e9-4fb3-8fc9-1939228873c0","Type":"ContainerStarted","Data":"a7dcf51aefbf51ecb189ed5a42e1920f1342dab38058da24c297a9a6d798d7c3"}
Sep 15 22:51:07.661716 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:07.661668    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4" podStartSLOduration=1.6616530840000001 podStartE2EDuration="1.661653084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:51:07.659716464 +0000 UTC m=+1440.278531371" watchObservedRunningTime="2026-09-15 22:51:07.661653084 +0000 UTC m=+1440.280467991"
Sep 15 22:51:08.528696 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:08.528665    2584 scope.go:117] "RemoveContainer" containerID="54326748c6408c92cf06f7416618878d2f6eaf54352b06bb01a67934da17e5ab"
Sep 15 22:51:08.537763 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:08.537743    2584 scope.go:117] "RemoveContainer" containerID="73bb5fa8a52a26772dcf86171b8c7dd60ce8476469e95d1d55c66af2377eeb97"
Sep 15 22:51:08.546003 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:08.545982    2584 scope.go:117] "RemoveContainer" containerID="9d10469c430757fa28fdd71838e806c97fe7365447aa61582e92ee7c501db077"
Sep 15 22:51:08.583937 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:08.583897    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:08.712010 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:08.711981    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:08.862708 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:08.862678    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:09.385684 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:09.385649    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:09.486046 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:09.485991    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:09.582664 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:09.582630    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:10.834446 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:10.834413    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:12.596908 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.596884    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:51:12.673011 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.672932    2584 generic.go:356] "Generic (PLEG): container finished" podID="40d601f3-2361-4855-8090-643aa8582094" containerID="c69a9b8174db96c58ad145b6eee1864d60549d45753c682a2f367446fb45e648" exitCode=0
Sep 15 22:51:12.673011 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.672985    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"
Sep 15 22:51:12.673192 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.673010    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z" event={"ID":"40d601f3-2361-4855-8090-643aa8582094","Type":"ContainerDied","Data":"c69a9b8174db96c58ad145b6eee1864d60549d45753c682a2f367446fb45e648"}
Sep 15 22:51:12.673192 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.673060    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z" event={"ID":"40d601f3-2361-4855-8090-643aa8582094","Type":"ContainerDied","Data":"8623b1a0d21febe7b9fbd00e82e88e0c5baf0ab03d214ecca4f78a37c005fc79"}
Sep 15 22:51:12.673192 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.673082    2584 scope.go:117] "RemoveContainer" containerID="c69a9b8174db96c58ad145b6eee1864d60549d45753c682a2f367446fb45e648"
Sep 15 22:51:12.682322 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.682301    2584 scope.go:117] "RemoveContainer" containerID="a8804954121d4ef8498ce4105583843bf52bb372f535d788866e20448df421cd"
Sep 15 22:51:12.690519 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.690502    2584 scope.go:117] "RemoveContainer" containerID="c69a9b8174db96c58ad145b6eee1864d60549d45753c682a2f367446fb45e648"
Sep 15 22:51:12.690788 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:51:12.690766    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69a9b8174db96c58ad145b6eee1864d60549d45753c682a2f367446fb45e648\": container with ID starting with c69a9b8174db96c58ad145b6eee1864d60549d45753c682a2f367446fb45e648 not found: ID does not exist" containerID="c69a9b8174db96c58ad145b6eee1864d60549d45753c682a2f367446fb45e648"
Sep 15 22:51:12.690838 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.690798    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69a9b8174db96c58ad145b6eee1864d60549d45753c682a2f367446fb45e648"} err="failed to get container status \"c69a9b8174db96c58ad145b6eee1864d60549d45753c682a2f367446fb45e648\": rpc error: code = NotFound desc = could not find container \"c69a9b8174db96c58ad145b6eee1864d60549d45753c682a2f367446fb45e648\": container with ID starting with c69a9b8174db96c58ad145b6eee1864d60549d45753c682a2f367446fb45e648 not found: ID does not exist"
Sep 15 22:51:12.690838 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.690815    2584 scope.go:117] "RemoveContainer" containerID="a8804954121d4ef8498ce4105583843bf52bb372f535d788866e20448df421cd"
Sep 15 22:51:12.691069 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:51:12.691052    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8804954121d4ef8498ce4105583843bf52bb372f535d788866e20448df421cd\": container with ID starting with a8804954121d4ef8498ce4105583843bf52bb372f535d788866e20448df421cd not found: ID does not exist" containerID="a8804954121d4ef8498ce4105583843bf52bb372f535d788866e20448df421cd"
Sep 15 22:51:12.691117 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.691076    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8804954121d4ef8498ce4105583843bf52bb372f535d788866e20448df421cd"} err="failed to get container status \"a8804954121d4ef8498ce4105583843bf52bb372f535d788866e20448df421cd\": rpc error: code = NotFound desc = could not find container \"a8804954121d4ef8498ce4105583843bf52bb372f535d788866e20448df421cd\": container with ID starting with a8804954121d4ef8498ce4105583843bf52bb372f535d788866e20448df421cd not found: ID does not exist"
Sep 15 22:51:12.692376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692355    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-credential-socket\") pod \"40d601f3-2361-4855-8090-643aa8582094\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") "
Sep 15 22:51:12.692481 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692394    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d601f3-2361-4855-8090-643aa8582094-istio-podinfo\") pod \"40d601f3-2361-4855-8090-643aa8582094\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") "
Sep 15 22:51:12.692481 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692414    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-workload-socket\") pod \"40d601f3-2361-4855-8090-643aa8582094\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") "
Sep 15 22:51:12.692481 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692453    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d601f3-2361-4855-8090-643aa8582094-istiod-ca-cert\") pod \"40d601f3-2361-4855-8090-643aa8582094\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") "
Sep 15 22:51:12.692607 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692568    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-istio-envoy\") pod \"40d601f3-2361-4855-8090-643aa8582094\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") "
Sep 15 22:51:12.692660 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692641    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d601f3-2361-4855-8090-643aa8582094-istio-token\") pod \"40d601f3-2361-4855-8090-643aa8582094\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") "
Sep 15 22:51:12.692721 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692642    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "40d601f3-2361-4855-8090-643aa8582094" (UID: "40d601f3-2361-4855-8090-643aa8582094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:51:12.692721 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692684    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-workload-certs\") pod \"40d601f3-2361-4855-8090-643aa8582094\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") "
Sep 15 22:51:12.692830 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692732    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-istio-data\") pod \"40d601f3-2361-4855-8090-643aa8582094\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") "
Sep 15 22:51:12.692830 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692750    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "40d601f3-2361-4855-8090-643aa8582094" (UID: "40d601f3-2361-4855-8090-643aa8582094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:51:12.692830 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692795    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwf7w\" (UniqueName: \"kubernetes.io/projected/40d601f3-2361-4855-8090-643aa8582094-kube-api-access-mwf7w\") pod \"40d601f3-2361-4855-8090-643aa8582094\" (UID: \"40d601f3-2361-4855-8090-643aa8582094\") "
Sep 15 22:51:12.692830 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692801    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d601f3-2361-4855-8090-643aa8582094-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "40d601f3-2361-4855-8090-643aa8582094" (UID: "40d601f3-2361-4855-8090-643aa8582094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:51:12.693053 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692955    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "40d601f3-2361-4855-8090-643aa8582094" (UID: "40d601f3-2361-4855-8090-643aa8582094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:51:12.693053 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.692988    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-istio-data" (OuterVolumeSpecName: "istio-data") pod "40d601f3-2361-4855-8090-643aa8582094" (UID: "40d601f3-2361-4855-8090-643aa8582094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:51:12.693279 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.693244    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-credential-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:51:12.693398 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.693287    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-workload-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:51:12.693398 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.693298    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d601f3-2361-4855-8090-643aa8582094-istiod-ca-cert\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:51:12.693398 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.693307    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-workload-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:51:12.693398 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.693319    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-istio-data\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:51:12.694474 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.694456    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40d601f3-2361-4855-8090-643aa8582094-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "40d601f3-2361-4855-8090-643aa8582094" (UID: "40d601f3-2361-4855-8090-643aa8582094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 22:51:12.694859 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.694838    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d601f3-2361-4855-8090-643aa8582094-istio-token" (OuterVolumeSpecName: "istio-token") pod "40d601f3-2361-4855-8090-643aa8582094" (UID: "40d601f3-2361-4855-8090-643aa8582094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:51:12.695058 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.695037    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "40d601f3-2361-4855-8090-643aa8582094" (UID: "40d601f3-2361-4855-8090-643aa8582094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:51:12.695058 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.695046    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d601f3-2361-4855-8090-643aa8582094-kube-api-access-mwf7w" (OuterVolumeSpecName: "kube-api-access-mwf7w") pod "40d601f3-2361-4855-8090-643aa8582094" (UID: "40d601f3-2361-4855-8090-643aa8582094"). InnerVolumeSpecName "kube-api-access-mwf7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:51:12.793787 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.793761    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwf7w\" (UniqueName: \"kubernetes.io/projected/40d601f3-2361-4855-8090-643aa8582094-kube-api-access-mwf7w\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:51:12.793787 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.793786    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d601f3-2361-4855-8090-643aa8582094-istio-podinfo\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:51:12.793945 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.793796    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d601f3-2361-4855-8090-643aa8582094-istio-envoy\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:51:12.793945 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.793805    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d601f3-2361-4855-8090-643aa8582094-istio-token\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:51:12.998136 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:12.998108    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"]
Sep 15 22:51:13.002182 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:13.002156    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6cdcda22-openshift-default-6b75f9444d-b4z5z"]
Sep 15 22:51:13.899884 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:13.899847    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d601f3-2361-4855-8090-643aa8582094" path="/var/lib/kubelet/pods/40d601f3-2361-4855-8090-643aa8582094/volumes"
Sep 15 22:51:16.574287 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:16.574223    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:16.674790 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:16.674757    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:16.859480 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:16.859407    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:16.859480 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:16.859471    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:16.872672 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:16.872649    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:17.332908 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:17.332876    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:17.714504 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:17.714422    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:51:23.283874 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:23.283837    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:23.384187 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:23.384147    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:23.483369 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:23.483336    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:27.672928 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:27.672890    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:33.953761 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:33.953725    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:34.196646 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:34.196614    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:37.063945 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:37.063908    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:51:49.429504 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:49.429465    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:55.105810 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:55.105772    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:57.361569 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.361534    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"]
Sep 15 22:51:57.362013 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.361998    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d601f3-2361-4855-8090-643aa8582094" containerName="istio-proxy"
Sep 15 22:51:57.362066 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.362015    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d601f3-2361-4855-8090-643aa8582094" containerName="istio-proxy"
Sep 15 22:51:57.362066 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.362033    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d601f3-2361-4855-8090-643aa8582094" containerName="istio-proxy"
Sep 15 22:51:57.362066 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.362039    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d601f3-2361-4855-8090-643aa8582094" containerName="istio-proxy"
Sep 15 22:51:57.362171 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.362113    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d601f3-2361-4855-8090-643aa8582094" containerName="istio-proxy"
Sep 15 22:51:57.362171 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.362124    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d601f3-2361-4855-8090-643aa8582094" containerName="istio-proxy"
Sep 15 22:51:57.365375 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.365355    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.369479 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.369450    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-92482b\"/\"default-dockercfg-nckqq\""
Sep 15 22:51:57.369604 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.369478    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-92482b\"/\"openshift-service-ca.crt\""
Sep 15 22:51:57.369651 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.369641    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-92482b\"/\"auth-test-model-92482b-kserve-self-signed-certs\""
Sep 15 22:51:57.369733 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.369716    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-92482b\"/\"kube-root-ca.crt\""
Sep 15 22:51:57.375312 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.375290    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"]
Sep 15 22:51:57.383230 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.383207    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:51:57.430258 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.430228    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-dshm\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.430396 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.430301    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzmtw\" (UniqueName: \"kubernetes.io/projected/1458f10f-2bfc-448f-b5c0-383aff19e70b-kube-api-access-rzmtw\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.430396 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.430379    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1458f10f-2bfc-448f-b5c0-383aff19e70b-tls-certs\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.430482 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.430426    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-model-cache\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.430527 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.430482    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-home\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.430527 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.430508    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-tmp-dir\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.531312 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.531283    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-model-cache\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.531431 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.531354    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-home\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.531431 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.531392    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-tmp-dir\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.531518 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.531453    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-dshm\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.531518 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.531503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzmtw\" (UniqueName: \"kubernetes.io/projected/1458f10f-2bfc-448f-b5c0-383aff19e70b-kube-api-access-rzmtw\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.531600 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.531559    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1458f10f-2bfc-448f-b5c0-383aff19e70b-tls-certs\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.531670 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.531652    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-model-cache\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.531725 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.531705    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-home\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.531777 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.531758    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-tmp-dir\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.533699 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.533675    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-dshm\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.534026 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.534009    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1458f10f-2bfc-448f-b5c0-383aff19e70b-tls-certs\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.540779 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.540753    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzmtw\" (UniqueName: \"kubernetes.io/projected/1458f10f-2bfc-448f-b5c0-383aff19e70b-kube-api-access-rzmtw\") pod \"auth-test-model-92482b-kserve-5b8865f85f-v8krm\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") " pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.676684 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.676602    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:51:57.951379 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:57.951347    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"]
Sep 15 22:51:57.952247 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:51:57.952221    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1458f10f_2bfc_448f_b5c0_383aff19e70b.slice/crio-b174c0473fd1250d555baceb486178b480eb9922784c72dd4e9ae44675006d24 WatchSource:0}: Error finding container b174c0473fd1250d555baceb486178b480eb9922784c72dd4e9ae44675006d24: Status 404 returned error can't find the container with id b174c0473fd1250d555baceb486178b480eb9922784c72dd4e9ae44675006d24
Sep 15 22:51:58.949522 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:58.949484    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm" event={"ID":"1458f10f-2bfc-448f-b5c0-383aff19e70b","Type":"ContainerStarted","Data":"c3ca9f862a8175b5856d2d4ce2af5b1d329aab4e32072a3efecf00b9c76d577c"}
Sep 15 22:51:58.949925 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:58.949533    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm" event={"ID":"1458f10f-2bfc-448f-b5c0-383aff19e70b","Type":"ContainerStarted","Data":"b174c0473fd1250d555baceb486178b480eb9922784c72dd4e9ae44675006d24"}
Sep 15 22:51:58.968175 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:51:58.968124    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm" podStartSLOduration=1.968108962 podStartE2EDuration="1.968108962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:51:58.967877309 +0000 UTC m=+1491.586692215" watchObservedRunningTime="2026-09-15 22:51:58.968108962 +0000 UTC m=+1491.586923870"
Sep 15 22:52:00.986387 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:00.986346    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:01.086915 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.086874    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:01.333775 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.333745    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:01.620710 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.620630    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"]
Sep 15 22:52:01.624538 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.624520    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.627386 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.627366    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-54db18-openshift-default-dockercfg-vqg2d\""
Sep 15 22:52:01.637477 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.637454    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"]
Sep 15 22:52:01.652567 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.652540    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:01.773122 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.773084    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-data\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.773122 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.773123    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-credential-socket\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.773415 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.773155    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-podinfo\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.773415 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.773277    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr964\" (UniqueName: \"kubernetes.io/projected/df60d562-b751-4ef5-a2fa-a962e6da07c3-kube-api-access-pr964\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.773415 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.773336    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-token\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.773415 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.773368    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df60d562-b751-4ef5-a2fa-a962e6da07c3-istiod-ca-cert\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.773415 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.773410    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-workload-certs\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.773664 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.773501    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-envoy\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.773664 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.773530    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-workload-socket\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.874894 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.874776    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr964\" (UniqueName: \"kubernetes.io/projected/df60d562-b751-4ef5-a2fa-a962e6da07c3-kube-api-access-pr964\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.874894 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.874842    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-token\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.874894 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.874874    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df60d562-b751-4ef5-a2fa-a962e6da07c3-istiod-ca-cert\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.875186 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.874915    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-workload-certs\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.875186 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.874978    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-envoy\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.875186 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.875008    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-workload-socket\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.875186 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.875066    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-data\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.875186 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.875097    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-credential-socket\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.875186 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.875141    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-podinfo\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.875611 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.875560    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-data\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.875611 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.875588    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-credential-socket\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.875751 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.875712    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-workload-certs\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.875939 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.875904    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df60d562-b751-4ef5-a2fa-a962e6da07c3-istiod-ca-cert\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.876093 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.875950    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-workload-socket\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.877986 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.877958    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-envoy\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.878315 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.878295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-podinfo\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.883396 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.883371    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-token\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.883976 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.883954    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr964\" (UniqueName: \"kubernetes.io/projected/df60d562-b751-4ef5-a2fa-a962e6da07c3-kube-api-access-pr964\") pod \"e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") " pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:01.936865 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:01.936835    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:02.191737 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:02.191642    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:02.216392 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:02.216359    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"]
Sep 15 22:52:02.217543 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:52:02.217521    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf60d562_b751_4ef5_a2fa_a962e6da07c3.slice/crio-5c74e2683a3a382d85f137ca82a7a57675a68caba530cc46b1b29e829ba7301a WatchSource:0}: Error finding container 5c74e2683a3a382d85f137ca82a7a57675a68caba530cc46b1b29e829ba7301a: Status 404 returned error can't find the container with id 5c74e2683a3a382d85f137ca82a7a57675a68caba530cc46b1b29e829ba7301a
Sep 15 22:52:02.220410 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:02.220376    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:52:02.220480 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:02.220441    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:52:02.220480 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:02.220469    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:52:02.979469 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:02.979427    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" event={"ID":"df60d562-b751-4ef5-a2fa-a962e6da07c3","Type":"ContainerStarted","Data":"3f1468baf5b24066b0ece8bc61f863b9698f4ea3dda5a8337c905e8e6533c8f6"}
Sep 15 22:52:02.979668 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:02.979486    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" event={"ID":"df60d562-b751-4ef5-a2fa-a962e6da07c3","Type":"ContainerStarted","Data":"5c74e2683a3a382d85f137ca82a7a57675a68caba530cc46b1b29e829ba7301a"}
Sep 15 22:52:03.003255 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:03.003203    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" podStartSLOduration=2.003188747 podStartE2EDuration="2.003188747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:52:03.000513513 +0000 UTC m=+1495.619328422" watchObservedRunningTime="2026-09-15 22:52:03.003188747 +0000 UTC m=+1495.622003665"
Sep 15 22:52:03.938075 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:03.938035    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:03.944368 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:03.944343    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:03.986160 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:03.986129    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:03.987230 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:03.987207    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:04.174944 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:04.174902    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:04.965077 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:04.965039    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:05.082511 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:05.082471    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:05.809942 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:05.809909    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:06.667004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:06.666972    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:06.863827 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:06.863784    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:52:07.450490 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:07.450444    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:07.677500 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:07.677466    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:52:07.677920 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:07.677514    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:52:07.690498 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:07.690471    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:52:07.955384 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:07.955357    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:52:07.957609 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:07.957584    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:52:08.020910 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:08.020877    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:52:08.160385 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:08.160352    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:08.575666 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:08.575634    2584 scope.go:117] "RemoveContainer" containerID="7bdf3b990b1fae43c1ebc07301bd3bc307e9f857272e542f88f808e814880f7e"
Sep 15 22:52:09.065113 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:09.065059    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:09.791702 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:09.791664    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:09.894669 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:09.894631    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:09.993562 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:09.993526    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:10.304515 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.304458    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:10.406081 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.406047    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:10.415110 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.415063    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"]
Sep 15 22:52:10.420379 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.420358    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"
Sep 15 22:52:10.423125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.423107    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-54db18\""
Sep 15 22:52:10.423232 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.423143    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-54db18-dockercfg-hkt49\""
Sep 15 22:52:10.431098 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.431078    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"]
Sep 15 22:52:10.563545 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.563445    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fclcz\" (UniqueName: \"kubernetes.io/projected/db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b-kube-api-access-fclcz\") pod \"payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm\" (UID: \"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"
Sep 15 22:52:10.563723 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.563616    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm\" (UID: \"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"
Sep 15 22:52:10.664751 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.664713    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm\" (UID: \"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"
Sep 15 22:52:10.664933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.664767    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fclcz\" (UniqueName: \"kubernetes.io/projected/db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b-kube-api-access-fclcz\") pod \"payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm\" (UID: \"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"
Sep 15 22:52:10.665503 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.665479    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm\" (UID: \"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"
Sep 15 22:52:10.675025 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.674996    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fclcz\" (UniqueName: \"kubernetes.io/projected/db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b-kube-api-access-fclcz\") pod \"payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm\" (UID: \"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"
Sep 15 22:52:10.732286 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:10.732237    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"
Sep 15 22:52:11.007026 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:11.007001    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"]
Sep 15 22:52:11.008763 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:52:11.008735    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4c3ea6_0bdd_491f_8a39_8f415ab4ca3b.slice/crio-9eb34c0fefa00f7a50481847b7dc1d578598bfbbed095deae095dd445bc7df50 WatchSource:0}: Error finding container 9eb34c0fefa00f7a50481847b7dc1d578598bfbbed095deae095dd445bc7df50: Status 404 returned error can't find the container with id 9eb34c0fefa00f7a50481847b7dc1d578598bfbbed095deae095dd445bc7df50
Sep 15 22:52:11.028603 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:11.028572    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm" event={"ID":"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b","Type":"ContainerStarted","Data":"9eb34c0fefa00f7a50481847b7dc1d578598bfbbed095deae095dd445bc7df50"}
Sep 15 22:52:12.036206 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:12.036161    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm" event={"ID":"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b","Type":"ContainerStarted","Data":"b84aa2869fe1f88c99bd2f197fd57e8d84c077041f728127a9d25bcec74e53b1"}
Sep 15 22:52:12.036612 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:12.036249    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"
Sep 15 22:52:12.053859 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:12.053814    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm" podStartSLOduration=2.053801598 podStartE2EDuration="2.053801598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:52:12.053404217 +0000 UTC m=+1504.672219124" watchObservedRunningTime="2026-09-15 22:52:12.053801598 +0000 UTC m=+1504.672616550"
Sep 15 22:52:23.044095 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:23.044060    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"
Sep 15 22:52:26.751940 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:26.751893    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:31.859833 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:31.859745    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:32.653094 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:32.653053    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:33.479991 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:33.479957    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:34.234985 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:34.234947    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:34.957541 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:34.957491    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:35.186861 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:35.186825    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:35.400508 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:35.400474    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:35.699882 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:35.699790    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:36.158055 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:36.158018    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:36.235672 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:36.235638    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:37.161650 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:37.161608    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:37.884604 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:37.884563    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:38.773122 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:38.773088    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:39.019999 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:39.019951    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:39.118840 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:39.118806    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:39.691069 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:39.691035    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:40.204295 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:40.204249    2584 generic.go:356] "Generic (PLEG): container finished" podID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerID="3f1468baf5b24066b0ece8bc61f863b9698f4ea3dda5a8337c905e8e6533c8f6" exitCode=0
Sep 15 22:52:40.204763 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:40.204332    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" event={"ID":"df60d562-b751-4ef5-a2fa-a962e6da07c3","Type":"ContainerDied","Data":"3f1468baf5b24066b0ece8bc61f863b9698f4ea3dda5a8337c905e8e6533c8f6"}
Sep 15 22:52:40.204832 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:40.204795    2584 scope.go:117] "RemoveContainer" containerID="3f1468baf5b24066b0ece8bc61f863b9698f4ea3dda5a8337c905e8e6533c8f6"
Sep 15 22:52:40.206102 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:40.206074    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:52:40.206156 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:40.206146    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:52:40.206211 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:40.206199    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:52:40.644006 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:40.643963    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:41.212553 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:41.212523    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" event={"ID":"df60d562-b751-4ef5-a2fa-a962e6da07c3","Type":"ContainerStarted","Data":"6c8300f89b4d4e62701fc000a863f6703c9e2cb7db2a0f4a259aa22d56648e0f"}
Sep 15 22:52:41.463220 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:41.463123    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:41.936994 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:41.936959    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:41.937169 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:41.937138    2584 patch_prober.go:28] interesting pod/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 15 22:52:41.937221 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:41.937197    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 15 22:52:42.343606 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:42.343567    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:42.937742 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:42.937708    2584 patch_prober.go:28] interesting pod/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 15 22:52:42.937901 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:42.937775    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 15 22:52:43.378773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:43.378738    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:43.938071 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:43.938035    2584 patch_prober.go:28] interesting pod/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 15 22:52:43.938243 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:43.938087    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 15 22:52:44.296056 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:44.295960    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:44.938027 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:44.937986    2584 patch_prober.go:28] interesting pod/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 15 22:52:44.938416 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:44.938064    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 15 22:52:45.092983 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:45.092953    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:45.937607 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:45.937570    2584 patch_prober.go:28] interesting pod/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 15 22:52:45.937807 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:45.937631    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 15 22:52:46.067172 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.067118    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:46.167884 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.167840    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:46.178682 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.178647    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:52:46.267664 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.267552    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:46.611576 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.611543    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:46.720749 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.720714    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:46.743690 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.743661    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"]
Sep 15 22:52:46.748836 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.748811    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"
Sep 15 22:52:46.751538 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.751516    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f667ee\""
Sep 15 22:52:46.751750 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.751715    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f667ee-dockercfg-s9jt7\""
Sep 15 22:52:46.760239 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.760216    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"]
Sep 15 22:52:46.931387 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.931295    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4693f3-2da2-4bf0-b64c-dc9b17657b8b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx\" (UID: \"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"
Sep 15 22:52:46.931387 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.931353    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcd82\" (UniqueName: \"kubernetes.io/projected/dc4693f3-2da2-4bf0-b64c-dc9b17657b8b-kube-api-access-gcd82\") pod \"payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx\" (UID: \"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"
Sep 15 22:52:46.937195 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.937159    2584 patch_prober.go:28] interesting pod/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 15 22:52:46.937350 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:46.937223    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 15 22:52:47.032813 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:47.032774    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4693f3-2da2-4bf0-b64c-dc9b17657b8b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx\" (UID: \"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"
Sep 15 22:52:47.033017 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:47.032826    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcd82\" (UniqueName: \"kubernetes.io/projected/dc4693f3-2da2-4bf0-b64c-dc9b17657b8b-kube-api-access-gcd82\") pod \"payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx\" (UID: \"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"
Sep 15 22:52:47.033707 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:47.033680    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4693f3-2da2-4bf0-b64c-dc9b17657b8b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx\" (UID: \"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"
Sep 15 22:52:47.049385 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:47.049356    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcd82\" (UniqueName: \"kubernetes.io/projected/dc4693f3-2da2-4bf0-b64c-dc9b17657b8b-kube-api-access-gcd82\") pod \"payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx\" (UID: \"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"
Sep 15 22:52:47.061412 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:47.061389    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"
Sep 15 22:52:47.164319 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:47.164282    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:47.359713 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:47.359681    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"]
Sep 15 22:52:47.361446 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:52:47.361417    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4693f3_2da2_4bf0_b64c_dc9b17657b8b.slice/crio-429fab5880c3aad65a7251c0d30f3fa5e2324f4ad5a7c7f3c389c69e677e7a31 WatchSource:0}: Error finding container 429fab5880c3aad65a7251c0d30f3fa5e2324f4ad5a7c7f3c389c69e677e7a31: Status 404 returned error can't find the container with id 429fab5880c3aad65a7251c0d30f3fa5e2324f4ad5a7c7f3c389c69e677e7a31
Sep 15 22:52:47.960371 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:47.960331    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:47.960843 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:47.960797    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:47.961956 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:47.961936    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:52:48.029933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:48.029888    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:52:48.262581 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:48.262474    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx" event={"ID":"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b","Type":"ContainerStarted","Data":"d124b51edca6ad86cc35b512576240daf23415a91c6cd78d491f8f653299ebdd"}
Sep 15 22:52:48.262581 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:48.262546    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx" event={"ID":"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b","Type":"ContainerStarted","Data":"429fab5880c3aad65a7251c0d30f3fa5e2324f4ad5a7c7f3c389c69e677e7a31"}
Sep 15 22:52:48.263080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:48.262941    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"
Sep 15 22:52:48.283033 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:48.282981    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx" podStartSLOduration=2.282961288 podStartE2EDuration="2.282961288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:52:48.282505658 +0000 UTC m=+1540.901320562" watchObservedRunningTime="2026-09-15 22:52:48.282961288 +0000 UTC m=+1540.901776202"
Sep 15 22:52:58.141563 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:52:58.141528    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:00.276139 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:00.276104    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"
Sep 15 22:53:08.252407 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:08.252373    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:08.603388 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:08.603357    2584 scope.go:117] "RemoveContainer" containerID="b50cca19dede60109d9276da15e4badeae6e610f5041d193e98a019504598b63"
Sep 15 22:53:09.063020 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:09.062988    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:09.181752 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:09.181721    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:09.869699 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:09.869652    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:10.682063 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:10.682019    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:11.423235 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:11.423186    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:11.953318 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:11.953281    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8"]
Sep 15 22:53:11.956336 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:11.956316    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8"
Sep 15 22:53:11.959031 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:11.959007    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:53:11.959148 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:11.959007    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:53:11.965202 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:11.965175    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8"]
Sep 15 22:53:12.084441 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:12.084402    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a601ac-830a-432b-bddc-a0fcf40a72ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-54db18-fljt8\" (UID: \"f6a601ac-830a-432b-bddc-a0fcf40a72ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8"
Sep 15 22:53:12.185680 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:12.185646    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a601ac-830a-432b-bddc-a0fcf40a72ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-54db18-fljt8\" (UID: \"f6a601ac-830a-432b-bddc-a0fcf40a72ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8"
Sep 15 22:53:12.186276 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:12.186246    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a601ac-830a-432b-bddc-a0fcf40a72ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-54db18-fljt8\" (UID: \"f6a601ac-830a-432b-bddc-a0fcf40a72ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8"
Sep 15 22:53:12.268712 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:12.268644    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8"
Sep 15 22:53:12.369149 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:12.368984    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:12.547386 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:12.545854    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8"]
Sep 15 22:53:12.554846 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:53:12.554809    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a601ac_830a_432b_bddc_a0fcf40a72ff.slice/crio-c45caac460685917dd493bb3365a41793539de8681724db3f8654053397c93f6 WatchSource:0}: Error finding container c45caac460685917dd493bb3365a41793539de8681724db3f8654053397c93f6: Status 404 returned error can't find the container with id c45caac460685917dd493bb3365a41793539de8681724db3f8654053397c93f6
Sep 15 22:53:13.368323 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:13.368286    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:13.407522 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:13.407495    2584 generic.go:356] "Generic (PLEG): container finished" podID="f6a601ac-830a-432b-bddc-a0fcf40a72ff" containerID="530b160b9073e40a883fb7fbd8dd7fa7b4ae643e7b68a6df031392e67ae8489e" exitCode=0
Sep 15 22:53:13.407662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:13.407552    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8" event={"ID":"f6a601ac-830a-432b-bddc-a0fcf40a72ff","Type":"ContainerDied","Data":"530b160b9073e40a883fb7fbd8dd7fa7b4ae643e7b68a6df031392e67ae8489e"}
Sep 15 22:53:13.407662 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:13.407606    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8" event={"ID":"f6a601ac-830a-432b-bddc-a0fcf40a72ff","Type":"ContainerStarted","Data":"c45caac460685917dd493bb3365a41793539de8681724db3f8654053397c93f6"}
Sep 15 22:53:14.099874 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:14.099835    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:14.556877 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:14.556850    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8"
Sep 15 22:53:14.711617 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:14.711537    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a601ac-830a-432b-bddc-a0fcf40a72ff-maas-api-service-ca\") pod \"f6a601ac-830a-432b-bddc-a0fcf40a72ff\" (UID: \"f6a601ac-830a-432b-bddc-a0fcf40a72ff\") "
Sep 15 22:53:14.711909 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:14.711887    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a601ac-830a-432b-bddc-a0fcf40a72ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6a601ac-830a-432b-bddc-a0fcf40a72ff" (UID: "f6a601ac-830a-432b-bddc-a0fcf40a72ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:53:14.812736 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:14.812699    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a601ac-830a-432b-bddc-a0fcf40a72ff-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:14.922589 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:14.922549    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:15.420179 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:15.420149    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8"
Sep 15 22:53:15.420674 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:15.420189    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8" event={"ID":"f6a601ac-830a-432b-bddc-a0fcf40a72ff","Type":"ContainerDied","Data":"c45caac460685917dd493bb3365a41793539de8681724db3f8654053397c93f6"}
Sep 15 22:53:15.420674 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:15.420240    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c45caac460685917dd493bb3365a41793539de8681724db3f8654053397c93f6"
Sep 15 22:53:15.637744 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:15.637714    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:16.452567 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:16.452528    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:17.284816 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:17.284783    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:18.010332 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:18.010296    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:18.881111 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:18.881074    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:22.097529 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:22.097490    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:25.865931 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:25.865890    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:53:27.848638 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:27.848598    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:27.883057 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:27.883023    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"]
Sep 15 22:53:27.883306 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:27.883277    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm" podUID="db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b" containerName="payload-pre-processing" containerID="cri-o://b84aa2869fe1f88c99bd2f197fd57e8d84c077041f728127a9d25bcec74e53b1" gracePeriod=30
Sep 15 22:53:27.920147 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:27.920109    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:28.020335 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.020257    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:28.049569 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.049404    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8"]
Sep 15 22:53:28.053289 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.053247    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54db18-fljt8"]
Sep 15 22:53:28.120690 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.120553    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:28.153939 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.151203    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"
Sep 15 22:53:28.245828 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.245792    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fclcz\" (UniqueName: \"kubernetes.io/projected/db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b-kube-api-access-fclcz\") pod \"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b\" (UID: \"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b\") "
Sep 15 22:53:28.246010 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.245962    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b-plugins-config-volume\") pod \"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b\" (UID: \"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b\") "
Sep 15 22:53:28.246369 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.246342    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b" (UID: "db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:53:28.247928 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.247904    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b-kube-api-access-fclcz" (OuterVolumeSpecName: "kube-api-access-fclcz") pod "db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b" (UID: "db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b"). InnerVolumeSpecName "kube-api-access-fclcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:53:28.347298 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.347253    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:28.347298 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.347295    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fclcz\" (UniqueName: \"kubernetes.io/projected/db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b-kube-api-access-fclcz\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:28.388969 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.388871    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:28.498910 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.498869    2584 generic.go:356] "Generic (PLEG): container finished" podID="db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b" containerID="b84aa2869fe1f88c99bd2f197fd57e8d84c077041f728127a9d25bcec74e53b1" exitCode=0
Sep 15 22:53:28.499120 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.498910    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm" event={"ID":"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b","Type":"ContainerDied","Data":"b84aa2869fe1f88c99bd2f197fd57e8d84c077041f728127a9d25bcec74e53b1"}
Sep 15 22:53:28.499120 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.498919    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"
Sep 15 22:53:28.499120 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.498973    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm" event={"ID":"db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b","Type":"ContainerDied","Data":"9eb34c0fefa00f7a50481847b7dc1d578598bfbbed095deae095dd445bc7df50"}
Sep 15 22:53:28.499120 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.498998    2584 scope.go:117] "RemoveContainer" containerID="b84aa2869fe1f88c99bd2f197fd57e8d84c077041f728127a9d25bcec74e53b1"
Sep 15 22:53:28.516693 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.516670    2584 scope.go:117] "RemoveContainer" containerID="b84aa2869fe1f88c99bd2f197fd57e8d84c077041f728127a9d25bcec74e53b1"
Sep 15 22:53:28.517005 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:53:28.516984    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b84aa2869fe1f88c99bd2f197fd57e8d84c077041f728127a9d25bcec74e53b1\": container with ID starting with b84aa2869fe1f88c99bd2f197fd57e8d84c077041f728127a9d25bcec74e53b1 not found: ID does not exist" containerID="b84aa2869fe1f88c99bd2f197fd57e8d84c077041f728127a9d25bcec74e53b1"
Sep 15 22:53:28.517074 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.517016    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84aa2869fe1f88c99bd2f197fd57e8d84c077041f728127a9d25bcec74e53b1"} err="failed to get container status \"b84aa2869fe1f88c99bd2f197fd57e8d84c077041f728127a9d25bcec74e53b1\": rpc error: code = NotFound desc = could not find container \"b84aa2869fe1f88c99bd2f197fd57e8d84c077041f728127a9d25bcec74e53b1\": container with ID starting with b84aa2869fe1f88c99bd2f197fd57e8d84c077041f728127a9d25bcec74e53b1 not found: ID does not exist"
Sep 15 22:53:28.529087 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.529061    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"]
Sep 15 22:53:28.535007 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.534985    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54db18-7cddcbf978-fkfnm"]
Sep 15 22:53:28.573475 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:28.573446    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:29.335163 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:29.335129    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:29.900837 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:29.900797    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b" path="/var/lib/kubelet/pods/db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b/volumes"
Sep 15 22:53:29.901245 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:29.901224    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6a601ac-830a-432b-bddc-a0fcf40a72ff" path="/var/lib/kubelet/pods/f6a601ac-830a-432b-bddc-a0fcf40a72ff/volumes"
Sep 15 22:53:30.251158 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:30.251070    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:30.925343 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:30.925306    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:31.858012 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:31.857975    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:32.763810 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:32.763774    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:33.471065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:33.471032    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:34.253850 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.253813    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:34.358688 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.358643    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:34.469950 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.469914    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"]
Sep 15 22:53:34.470227 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.470200    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerName="istio-proxy" containerID="cri-o://6c8300f89b4d4e62701fc000a863f6703c9e2cb7db2a0f4a259aa22d56648e0f" gracePeriod=30
Sep 15 22:53:34.491552 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.491525    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:34.817774 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.817740    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2"]
Sep 15 22:53:34.818313 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.818293    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a601ac-830a-432b-bddc-a0fcf40a72ff" containerName="revoke-keys"
Sep 15 22:53:34.818313 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.818314    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a601ac-830a-432b-bddc-a0fcf40a72ff" containerName="revoke-keys"
Sep 15 22:53:34.818485 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.818329    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b" containerName="payload-pre-processing"
Sep 15 22:53:34.818485 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.818337    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b" containerName="payload-pre-processing"
Sep 15 22:53:34.818485 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.818446    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a601ac-830a-432b-bddc-a0fcf40a72ff" containerName="revoke-keys"
Sep 15 22:53:34.818485 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.818463    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="db4c3ea6-0bdd-491f-8a39-8f415ab4ca3b" containerName="payload-pre-processing"
Sep 15 22:53:34.823086 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.823065    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2"
Sep 15 22:53:34.825930 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.825887    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:53:34.826167 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.826150    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:53:34.829159 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.829133    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2"]
Sep 15 22:53:34.913862 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:34.913826    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58868387-6bf1-4209-88f0-9bc69e560224-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2\" (UID: \"58868387-6bf1-4209-88f0-9bc69e560224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2"
Sep 15 22:53:35.015084 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:35.015044    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58868387-6bf1-4209-88f0-9bc69e560224-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2\" (UID: \"58868387-6bf1-4209-88f0-9bc69e560224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2"
Sep 15 22:53:35.015698 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:35.015676    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58868387-6bf1-4209-88f0-9bc69e560224-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2\" (UID: \"58868387-6bf1-4209-88f0-9bc69e560224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2"
Sep 15 22:53:35.135432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:35.135351    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2"
Sep 15 22:53:35.271383 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:35.271351    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:35.411246 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:35.411218    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2"]
Sep 15 22:53:35.412772 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:53:35.412750    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58868387_6bf1_4209_88f0_9bc69e560224.slice/crio-36f422c1cb11dd66e6a1293696d9093b57f5a81bb7361b5d9ca3074f1e59427a WatchSource:0}: Error finding container 36f422c1cb11dd66e6a1293696d9093b57f5a81bb7361b5d9ca3074f1e59427a: Status 404 returned error can't find the container with id 36f422c1cb11dd66e6a1293696d9093b57f5a81bb7361b5d9ca3074f1e59427a
Sep 15 22:53:35.540715 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:35.540684    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2" event={"ID":"58868387-6bf1-4209-88f0-9bc69e560224","Type":"ContainerStarted","Data":"36f422c1cb11dd66e6a1293696d9093b57f5a81bb7361b5d9ca3074f1e59427a"}
Sep 15 22:53:35.971376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:35.971293    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:36.548366 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:36.548326    2584 generic.go:356] "Generic (PLEG): container finished" podID="58868387-6bf1-4209-88f0-9bc69e560224" containerID="9c82be68e66d9c4e4f12f93b935110eadd9cc32b5ea0ad5de09d60fb953ee301" exitCode=0
Sep 15 22:53:36.548790 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:36.548422    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2" event={"ID":"58868387-6bf1-4209-88f0-9bc69e560224","Type":"ContainerDied","Data":"9c82be68e66d9c4e4f12f93b935110eadd9cc32b5ea0ad5de09d60fb953ee301"}
Sep 15 22:53:36.753250 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:36.753217    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:37.461157 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:37.461126    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:37.696007 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:37.695982    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2"
Sep 15 22:53:37.744768 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:37.744700    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58868387-6bf1-4209-88f0-9bc69e560224-maas-api-service-ca\") pod \"58868387-6bf1-4209-88f0-9bc69e560224\" (UID: \"58868387-6bf1-4209-88f0-9bc69e560224\") "
Sep 15 22:53:37.745083 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:37.745060    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58868387-6bf1-4209-88f0-9bc69e560224-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "58868387-6bf1-4209-88f0-9bc69e560224" (UID: "58868387-6bf1-4209-88f0-9bc69e560224"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:53:37.745173 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:37.745159    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58868387-6bf1-4209-88f0-9bc69e560224-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:38.258713 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:38.258678    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:38.561417 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:38.561379    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2" event={"ID":"58868387-6bf1-4209-88f0-9bc69e560224","Type":"ContainerDied","Data":"36f422c1cb11dd66e6a1293696d9093b57f5a81bb7361b5d9ca3074f1e59427a"}
Sep 15 22:53:38.561587 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:38.561415    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2"
Sep 15 22:53:38.561587 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:38.561468    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36f422c1cb11dd66e6a1293696d9093b57f5a81bb7361b5d9ca3074f1e59427a"
Sep 15 22:53:39.075654 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.075622    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:39.744016 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.743949    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:53:39.770025 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.769997    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:39.865147 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865111    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-data\") pod \"df60d562-b751-4ef5-a2fa-a962e6da07c3\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") "
Sep 15 22:53:39.865340 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865181    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-workload-certs\") pod \"df60d562-b751-4ef5-a2fa-a962e6da07c3\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") "
Sep 15 22:53:39.865340 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865227    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-workload-socket\") pod \"df60d562-b751-4ef5-a2fa-a962e6da07c3\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") "
Sep 15 22:53:39.865340 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865278    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df60d562-b751-4ef5-a2fa-a962e6da07c3-istiod-ca-cert\") pod \"df60d562-b751-4ef5-a2fa-a962e6da07c3\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") "
Sep 15 22:53:39.865340 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865314    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-credential-socket\") pod \"df60d562-b751-4ef5-a2fa-a962e6da07c3\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") "
Sep 15 22:53:39.865340 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865334    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-podinfo\") pod \"df60d562-b751-4ef5-a2fa-a962e6da07c3\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") "
Sep 15 22:53:39.865596 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865374    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-envoy\") pod \"df60d562-b751-4ef5-a2fa-a962e6da07c3\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") "
Sep 15 22:53:39.865596 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865398    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-data" (OuterVolumeSpecName: "istio-data") pod "df60d562-b751-4ef5-a2fa-a962e6da07c3" (UID: "df60d562-b751-4ef5-a2fa-a962e6da07c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:53:39.865596 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865479    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pr964\" (UniqueName: \"kubernetes.io/projected/df60d562-b751-4ef5-a2fa-a962e6da07c3-kube-api-access-pr964\") pod \"df60d562-b751-4ef5-a2fa-a962e6da07c3\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") "
Sep 15 22:53:39.865596 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865474    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df60d562-b751-4ef5-a2fa-a962e6da07c3" (UID: "df60d562-b751-4ef5-a2fa-a962e6da07c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:53:39.865596 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865523    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-token\") pod \"df60d562-b751-4ef5-a2fa-a962e6da07c3\" (UID: \"df60d562-b751-4ef5-a2fa-a962e6da07c3\") "
Sep 15 22:53:39.865596 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865549    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df60d562-b751-4ef5-a2fa-a962e6da07c3" (UID: "df60d562-b751-4ef5-a2fa-a962e6da07c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:53:39.865905 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865825    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df60d562-b751-4ef5-a2fa-a962e6da07c3" (UID: "df60d562-b751-4ef5-a2fa-a962e6da07c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:53:39.866009 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.865984    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-workload-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:39.866009 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.866010    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-workload-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:39.866186 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.866024    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-credential-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:39.866186 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.866037    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-data\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:39.866186 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.866060    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df60d562-b751-4ef5-a2fa-a962e6da07c3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df60d562-b751-4ef5-a2fa-a962e6da07c3" (UID: "df60d562-b751-4ef5-a2fa-a962e6da07c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:53:39.867777 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.867751    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-token" (OuterVolumeSpecName: "istio-token") pod "df60d562-b751-4ef5-a2fa-a962e6da07c3" (UID: "df60d562-b751-4ef5-a2fa-a962e6da07c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:53:39.867945 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.867912    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df60d562-b751-4ef5-a2fa-a962e6da07c3-kube-api-access-pr964" (OuterVolumeSpecName: "kube-api-access-pr964") pod "df60d562-b751-4ef5-a2fa-a962e6da07c3" (UID: "df60d562-b751-4ef5-a2fa-a962e6da07c3"). InnerVolumeSpecName "kube-api-access-pr964". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:53:39.868153 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.868136    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df60d562-b751-4ef5-a2fa-a962e6da07c3" (UID: "df60d562-b751-4ef5-a2fa-a962e6da07c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:53:39.868226 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.868185    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df60d562-b751-4ef5-a2fa-a962e6da07c3" (UID: "df60d562-b751-4ef5-a2fa-a962e6da07c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 22:53:39.966840 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.966815    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-envoy\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:39.966840 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.966841    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr964\" (UniqueName: \"kubernetes.io/projected/df60d562-b751-4ef5-a2fa-a962e6da07c3-kube-api-access-pr964\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:39.967006 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.966851    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-token\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:39.967006 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.966859    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df60d562-b751-4ef5-a2fa-a962e6da07c3-istiod-ca-cert\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:39.967006 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:39.966867    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df60d562-b751-4ef5-a2fa-a962e6da07c3-istio-podinfo\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:40.574356 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:40.574317    2584 generic.go:356] "Generic (PLEG): container finished" podID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerID="6c8300f89b4d4e62701fc000a863f6703c9e2cb7db2a0f4a259aa22d56648e0f" exitCode=0
Sep 15 22:53:40.574779 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:40.574366    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"
Sep 15 22:53:40.574779 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:40.574398    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" event={"ID":"df60d562-b751-4ef5-a2fa-a962e6da07c3","Type":"ContainerDied","Data":"6c8300f89b4d4e62701fc000a863f6703c9e2cb7db2a0f4a259aa22d56648e0f"}
Sep 15 22:53:40.574779 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:40.574450    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg" event={"ID":"df60d562-b751-4ef5-a2fa-a962e6da07c3","Type":"ContainerDied","Data":"5c74e2683a3a382d85f137ca82a7a57675a68caba530cc46b1b29e829ba7301a"}
Sep 15 22:53:40.574779 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:40.574473    2584 scope.go:117] "RemoveContainer" containerID="6c8300f89b4d4e62701fc000a863f6703c9e2cb7db2a0f4a259aa22d56648e0f"
Sep 15 22:53:40.584511 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:40.584492    2584 scope.go:117] "RemoveContainer" containerID="3f1468baf5b24066b0ece8bc61f863b9698f4ea3dda5a8337c905e8e6533c8f6"
Sep 15 22:53:40.585058 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:40.585038    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:40.593586 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:40.593173    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"]
Sep 15 22:53:40.593586 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:40.593284    2584 scope.go:117] "RemoveContainer" containerID="6c8300f89b4d4e62701fc000a863f6703c9e2cb7db2a0f4a259aa22d56648e0f"
Sep 15 22:53:40.593586 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:53:40.593565    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8300f89b4d4e62701fc000a863f6703c9e2cb7db2a0f4a259aa22d56648e0f\": container with ID starting with 6c8300f89b4d4e62701fc000a863f6703c9e2cb7db2a0f4a259aa22d56648e0f not found: ID does not exist" containerID="6c8300f89b4d4e62701fc000a863f6703c9e2cb7db2a0f4a259aa22d56648e0f"
Sep 15 22:53:40.593747 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:40.593610    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8300f89b4d4e62701fc000a863f6703c9e2cb7db2a0f4a259aa22d56648e0f"} err="failed to get container status \"6c8300f89b4d4e62701fc000a863f6703c9e2cb7db2a0f4a259aa22d56648e0f\": rpc error: code = NotFound desc = could not find container \"6c8300f89b4d4e62701fc000a863f6703c9e2cb7db2a0f4a259aa22d56648e0f\": container with ID starting with 6c8300f89b4d4e62701fc000a863f6703c9e2cb7db2a0f4a259aa22d56648e0f not found: ID does not exist"
Sep 15 22:53:40.593747 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:40.593633    2584 scope.go:117] "RemoveContainer" containerID="3f1468baf5b24066b0ece8bc61f863b9698f4ea3dda5a8337c905e8e6533c8f6"
Sep 15 22:53:40.593921 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:53:40.593903    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f1468baf5b24066b0ece8bc61f863b9698f4ea3dda5a8337c905e8e6533c8f6\": container with ID starting with 3f1468baf5b24066b0ece8bc61f863b9698f4ea3dda5a8337c905e8e6533c8f6 not found: ID does not exist" containerID="3f1468baf5b24066b0ece8bc61f863b9698f4ea3dda5a8337c905e8e6533c8f6"
Sep 15 22:53:40.593961 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:40.593926    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f1468baf5b24066b0ece8bc61f863b9698f4ea3dda5a8337c905e8e6533c8f6"} err="failed to get container status \"3f1468baf5b24066b0ece8bc61f863b9698f4ea3dda5a8337c905e8e6533c8f6\": rpc error: code = NotFound desc = could not find container \"3f1468baf5b24066b0ece8bc61f863b9698f4ea3dda5a8337c905e8e6533c8f6\": container with ID starting with 3f1468baf5b24066b0ece8bc61f863b9698f4ea3dda5a8337c905e8e6533c8f6 not found: ID does not exist"
Sep 15 22:53:40.599342 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:40.599320    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-54db18-openshift-default-8fbc7c8d4-b95rg"]
Sep 15 22:53:41.900676 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:41.900642    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" path="/var/lib/kubelet/pods/df60d562-b751-4ef5-a2fa-a962e6da07c3/volumes"
Sep 15 22:53:44.956958 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:44.956927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:50.556748 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.556712    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:50.614933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.614897    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"]
Sep 15 22:53:50.616149 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.616089    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx" podUID="dc4693f3-2da2-4bf0-b64c-dc9b17657b8b" containerName="payload-processing" containerID="cri-o://d124b51edca6ad86cc35b512576240daf23415a91c6cd78d491f8f653299ebdd" gracePeriod=30
Sep 15 22:53:50.642214 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.642131    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:50.701931 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.701893    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:50.770758 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.770722    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2"]
Sep 15 22:53:50.777307 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.777243    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f667ee-nzxb2"]
Sep 15 22:53:50.777991 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.777947    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:50.842489 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.842354    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:50.897071 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.897042    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"
Sep 15 22:53:50.979073 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.979038    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4693f3-2da2-4bf0-b64c-dc9b17657b8b-plugins-config-volume\") pod \"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b\" (UID: \"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b\") "
Sep 15 22:53:50.979253 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.979201    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcd82\" (UniqueName: \"kubernetes.io/projected/dc4693f3-2da2-4bf0-b64c-dc9b17657b8b-kube-api-access-gcd82\") pod \"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b\" (UID: \"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b\") "
Sep 15 22:53:50.979509 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.979477    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4693f3-2da2-4bf0-b64c-dc9b17657b8b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dc4693f3-2da2-4bf0-b64c-dc9b17657b8b" (UID: "dc4693f3-2da2-4bf0-b64c-dc9b17657b8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:53:50.979620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.979594    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4693f3-2da2-4bf0-b64c-dc9b17657b8b-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:50.981371 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:50.981349    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4693f3-2da2-4bf0-b64c-dc9b17657b8b-kube-api-access-gcd82" (OuterVolumeSpecName: "kube-api-access-gcd82") pod "dc4693f3-2da2-4bf0-b64c-dc9b17657b8b" (UID: "dc4693f3-2da2-4bf0-b64c-dc9b17657b8b"). InnerVolumeSpecName "kube-api-access-gcd82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:53:51.080892 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.080857    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcd82\" (UniqueName: \"kubernetes.io/projected/dc4693f3-2da2-4bf0-b64c-dc9b17657b8b-kube-api-access-gcd82\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:53:51.157510 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.157433    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:51.256984 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.256955    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:51.650168 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.650132    2584 generic.go:356] "Generic (PLEG): container finished" podID="dc4693f3-2da2-4bf0-b64c-dc9b17657b8b" containerID="d124b51edca6ad86cc35b512576240daf23415a91c6cd78d491f8f653299ebdd" exitCode=0
Sep 15 22:53:51.650670 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.650175    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"
Sep 15 22:53:51.650670 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.650216    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx" event={"ID":"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b","Type":"ContainerDied","Data":"d124b51edca6ad86cc35b512576240daf23415a91c6cd78d491f8f653299ebdd"}
Sep 15 22:53:51.650670 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.650308    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx" event={"ID":"dc4693f3-2da2-4bf0-b64c-dc9b17657b8b","Type":"ContainerDied","Data":"429fab5880c3aad65a7251c0d30f3fa5e2324f4ad5a7c7f3c389c69e677e7a31"}
Sep 15 22:53:51.650670 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.650345    2584 scope.go:117] "RemoveContainer" containerID="d124b51edca6ad86cc35b512576240daf23415a91c6cd78d491f8f653299ebdd"
Sep 15 22:53:51.659692 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.659673    2584 scope.go:117] "RemoveContainer" containerID="d124b51edca6ad86cc35b512576240daf23415a91c6cd78d491f8f653299ebdd"
Sep 15 22:53:51.659956 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:53:51.659940    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d124b51edca6ad86cc35b512576240daf23415a91c6cd78d491f8f653299ebdd\": container with ID starting with d124b51edca6ad86cc35b512576240daf23415a91c6cd78d491f8f653299ebdd not found: ID does not exist" containerID="d124b51edca6ad86cc35b512576240daf23415a91c6cd78d491f8f653299ebdd"
Sep 15 22:53:51.660009 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.659964    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d124b51edca6ad86cc35b512576240daf23415a91c6cd78d491f8f653299ebdd"} err="failed to get container status \"d124b51edca6ad86cc35b512576240daf23415a91c6cd78d491f8f653299ebdd\": rpc error: code = NotFound desc = could not find container \"d124b51edca6ad86cc35b512576240daf23415a91c6cd78d491f8f653299ebdd\": container with ID starting with d124b51edca6ad86cc35b512576240daf23415a91c6cd78d491f8f653299ebdd not found: ID does not exist"
Sep 15 22:53:51.679854 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.679831    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"]
Sep 15 22:53:51.683546 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.683525    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f667ee-cd6549c99-wz8mx"]
Sep 15 22:53:51.902418 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.902331    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58868387-6bf1-4209-88f0-9bc69e560224" path="/var/lib/kubelet/pods/58868387-6bf1-4209-88f0-9bc69e560224/volumes"
Sep 15 22:53:51.902857 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:51.902841    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc4693f3-2da2-4bf0-b64c-dc9b17657b8b" path="/var/lib/kubelet/pods/dc4693f3-2da2-4bf0-b64c-dc9b17657b8b/volumes"
Sep 15 22:53:52.055918 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:52.055889    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:52.774771 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:52.774736    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:53.753059 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:53.753027    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:54.458892 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:54.458856    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:55.161004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:55.160971    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:55.876981 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:55.876947    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:57.000636 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.000596    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:57.101607 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.101570    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:57.251027 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.250946    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:57.587031 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.586993    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz"]
Sep 15 22:53:57.587708 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.587686    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerName="istio-proxy"
Sep 15 22:53:57.587768 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.587713    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerName="istio-proxy"
Sep 15 22:53:57.587768 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.587731    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerName="istio-proxy"
Sep 15 22:53:57.587768 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.587740    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerName="istio-proxy"
Sep 15 22:53:57.587768 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.587757    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc4693f3-2da2-4bf0-b64c-dc9b17657b8b" containerName="payload-processing"
Sep 15 22:53:57.587768 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.587767    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc4693f3-2da2-4bf0-b64c-dc9b17657b8b" containerName="payload-processing"
Sep 15 22:53:57.587936 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.587809    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58868387-6bf1-4209-88f0-9bc69e560224" containerName="revoke-keys"
Sep 15 22:53:57.587936 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.587819    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="58868387-6bf1-4209-88f0-9bc69e560224" containerName="revoke-keys"
Sep 15 22:53:57.588002 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.587941    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerName="istio-proxy"
Sep 15 22:53:57.588002 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.587953    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc4693f3-2da2-4bf0-b64c-dc9b17657b8b" containerName="payload-processing"
Sep 15 22:53:57.588002 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.587965    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="df60d562-b751-4ef5-a2fa-a962e6da07c3" containerName="istio-proxy"
Sep 15 22:53:57.588002 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.587974    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="58868387-6bf1-4209-88f0-9bc69e560224" containerName="revoke-keys"
Sep 15 22:53:57.592597 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.592577    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz"
Sep 15 22:53:57.595715 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.595463    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:53:57.595715 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.595476    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:53:57.597993 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.597970    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz"]
Sep 15 22:53:57.749536 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.749500    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e9c13a-6949-435d-8df8-2813d66e7eb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz\" (UID: \"41e9c13a-6949-435d-8df8-2813d66e7eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz"
Sep 15 22:53:57.843830 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.843737    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:57.851071 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.851039    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e9c13a-6949-435d-8df8-2813d66e7eb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz\" (UID: \"41e9c13a-6949-435d-8df8-2813d66e7eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz"
Sep 15 22:53:57.851718 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.851697    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e9c13a-6949-435d-8df8-2813d66e7eb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz\" (UID: \"41e9c13a-6949-435d-8df8-2813d66e7eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz"
Sep 15 22:53:57.906112 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.906088    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz"
Sep 15 22:53:57.947336 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.947197    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:57.968239 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.968214    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj"]
Sep 15 22:53:57.972570 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.972329    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj"
Sep 15 22:53:57.985491 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:57.984665    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj"]
Sep 15 22:53:58.154864 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:58.154763    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eade4bb3-6bb0-4783-ad02-8456bf107b8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj\" (UID: \"eade4bb3-6bb0-4783-ad02-8456bf107b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj"
Sep 15 22:53:58.189955 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:58.189920    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz"]
Sep 15 22:53:58.191048 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:53:58.191004    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e9c13a_6949_435d_8df8_2813d66e7eb3.slice/crio-2ad23d88310e0fbf281f339808e6bfc60df1cc85550af7b27a64307dfab98730 WatchSource:0}: Error finding container 2ad23d88310e0fbf281f339808e6bfc60df1cc85550af7b27a64307dfab98730: Status 404 returned error can't find the container with id 2ad23d88310e0fbf281f339808e6bfc60df1cc85550af7b27a64307dfab98730
Sep 15 22:53:58.255324 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:58.255292    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eade4bb3-6bb0-4783-ad02-8456bf107b8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj\" (UID: \"eade4bb3-6bb0-4783-ad02-8456bf107b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj"
Sep 15 22:53:58.255867 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:58.255848    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eade4bb3-6bb0-4783-ad02-8456bf107b8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj\" (UID: \"eade4bb3-6bb0-4783-ad02-8456bf107b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj"
Sep 15 22:53:58.295601 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:58.295578    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj"
Sep 15 22:53:58.571743 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:58.571718    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj"]
Sep 15 22:53:58.573662 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:53:58.573636    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeade4bb3_6bb0_4783_ad02_8456bf107b8e.slice/crio-c10e831b28f9aadf7aa1587e2b7c6041658397e512f6413fe679809a12f39a83 WatchSource:0}: Error finding container c10e831b28f9aadf7aa1587e2b7c6041658397e512f6413fe679809a12f39a83: Status 404 returned error can't find the container with id c10e831b28f9aadf7aa1587e2b7c6041658397e512f6413fe679809a12f39a83
Sep 15 22:53:58.702462 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:58.702430    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj" event={"ID":"eade4bb3-6bb0-4783-ad02-8456bf107b8e","Type":"ContainerStarted","Data":"c10e831b28f9aadf7aa1587e2b7c6041658397e512f6413fe679809a12f39a83"}
Sep 15 22:53:58.704189 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:58.704156    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz" event={"ID":"41e9c13a-6949-435d-8df8-2813d66e7eb3","Type":"ContainerStarted","Data":"35efeef0846d16ba162786657bc7844f57b815e495ae4a7ca6805d5ae25741e1"}
Sep 15 22:53:58.704315 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:58.704198    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz" event={"ID":"41e9c13a-6949-435d-8df8-2813d66e7eb3","Type":"ContainerStarted","Data":"2ad23d88310e0fbf281f339808e6bfc60df1cc85550af7b27a64307dfab98730"}
Sep 15 22:53:58.720037 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:58.719928    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz" podStartSLOduration=1.7199125020000001 podStartE2EDuration="1.719912502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:53:58.719834607 +0000 UTC m=+1611.338649515" watchObservedRunningTime="2026-09-15 22:53:58.719912502 +0000 UTC m=+1611.338727411"
Sep 15 22:53:58.744135 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:58.744072    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:59.452796 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:59.452766    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:53:59.711829 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:59.711735    2584 generic.go:356] "Generic (PLEG): container finished" podID="41e9c13a-6949-435d-8df8-2813d66e7eb3" containerID="35efeef0846d16ba162786657bc7844f57b815e495ae4a7ca6805d5ae25741e1" exitCode=0
Sep 15 22:53:59.711829 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:59.711794    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz" event={"ID":"41e9c13a-6949-435d-8df8-2813d66e7eb3","Type":"ContainerDied","Data":"35efeef0846d16ba162786657bc7844f57b815e495ae4a7ca6805d5ae25741e1"}
Sep 15 22:53:59.713464 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:59.713443    2584 generic.go:356] "Generic (PLEG): container finished" podID="eade4bb3-6bb0-4783-ad02-8456bf107b8e" containerID="8c5ca628adc7b6781c88d946ff3733a677a408dee2ebde23f5517acabfae7ff7" exitCode=0
Sep 15 22:53:59.713578 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:53:59.713524    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj" event={"ID":"eade4bb3-6bb0-4783-ad02-8456bf107b8e","Type":"ContainerDied","Data":"8c5ca628adc7b6781c88d946ff3733a677a408dee2ebde23f5517acabfae7ff7"}
Sep 15 22:54:00.272701 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:00.272608    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:00.877115 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:00.877084    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj"
Sep 15 22:54:00.879872 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:00.879852    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz"
Sep 15 22:54:00.983860 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:00.983775    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eade4bb3-6bb0-4783-ad02-8456bf107b8e-maas-api-service-ca\") pod \"eade4bb3-6bb0-4783-ad02-8456bf107b8e\" (UID: \"eade4bb3-6bb0-4783-ad02-8456bf107b8e\") "
Sep 15 22:54:00.983860 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:00.983823    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e9c13a-6949-435d-8df8-2813d66e7eb3-maas-api-service-ca\") pod \"41e9c13a-6949-435d-8df8-2813d66e7eb3\" (UID: \"41e9c13a-6949-435d-8df8-2813d66e7eb3\") "
Sep 15 22:54:00.984166 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:00.984131    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eade4bb3-6bb0-4783-ad02-8456bf107b8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eade4bb3-6bb0-4783-ad02-8456bf107b8e" (UID: "eade4bb3-6bb0-4783-ad02-8456bf107b8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:54:00.984249 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:00.984207    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e9c13a-6949-435d-8df8-2813d66e7eb3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41e9c13a-6949-435d-8df8-2813d66e7eb3" (UID: "41e9c13a-6949-435d-8df8-2813d66e7eb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:54:00.992407 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:00.992382    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:01.085439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:01.085406    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e9c13a-6949-435d-8df8-2813d66e7eb3-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:01.085439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:01.085434    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eade4bb3-6bb0-4783-ad02-8456bf107b8e-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:01.726240 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:01.726211    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz"
Sep 15 22:54:01.726449 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:01.726257    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz" event={"ID":"41e9c13a-6949-435d-8df8-2813d66e7eb3","Type":"ContainerDied","Data":"2ad23d88310e0fbf281f339808e6bfc60df1cc85550af7b27a64307dfab98730"}
Sep 15 22:54:01.726449 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:01.726332    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ad23d88310e0fbf281f339808e6bfc60df1cc85550af7b27a64307dfab98730"
Sep 15 22:54:01.727658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:01.727642    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj"
Sep 15 22:54:01.727761 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:01.727682    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj" event={"ID":"eade4bb3-6bb0-4783-ad02-8456bf107b8e","Type":"ContainerDied","Data":"c10e831b28f9aadf7aa1587e2b7c6041658397e512f6413fe679809a12f39a83"}
Sep 15 22:54:01.727761 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:01.727723    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c10e831b28f9aadf7aa1587e2b7c6041658397e512f6413fe679809a12f39a83"
Sep 15 22:54:01.867815 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:01.867780    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:02.625954 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:02.625916    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:02.848821 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:02.848787    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"]
Sep 15 22:54:02.858203 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:02.858178    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"]
Sep 15 22:54:02.950367 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:02.950256    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:03.526595 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:03.526556    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:05.964577 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:05.964536    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:54:08.169156 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.169085    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:08.195401 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.195365    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"]
Sep 15 22:54:08.195678 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.195628    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54" podUID="73971904-e99b-4662-a71b-0ee3b26640d5" containerName="payload-processing" containerID="cri-o://99501850696cb9dd7b5210648c1cac6689b3e6366da91523833ca951abe27f30" gracePeriod=30
Sep 15 22:54:08.241159 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.241127    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:08.306664 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.306624    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:08.367447 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.367416    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz"]
Sep 15 22:54:08.370022 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.369996    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f2241f-5rlgz"]
Sep 15 22:54:08.370898 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.370871    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:08.445414 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.445385    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:08.459625 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.459599    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"
Sep 15 22:54:08.553991 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.553958    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdll6\" (UniqueName: \"kubernetes.io/projected/73971904-e99b-4662-a71b-0ee3b26640d5-kube-api-access-sdll6\") pod \"73971904-e99b-4662-a71b-0ee3b26640d5\" (UID: \"73971904-e99b-4662-a71b-0ee3b26640d5\") "
Sep 15 22:54:08.553991 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.553995    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73971904-e99b-4662-a71b-0ee3b26640d5-plugins-config-volume\") pod \"73971904-e99b-4662-a71b-0ee3b26640d5\" (UID: \"73971904-e99b-4662-a71b-0ee3b26640d5\") "
Sep 15 22:54:08.554473 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.554454    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73971904-e99b-4662-a71b-0ee3b26640d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73971904-e99b-4662-a71b-0ee3b26640d5" (UID: "73971904-e99b-4662-a71b-0ee3b26640d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:54:08.556033 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.556005    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73971904-e99b-4662-a71b-0ee3b26640d5-kube-api-access-sdll6" (OuterVolumeSpecName: "kube-api-access-sdll6") pod "73971904-e99b-4662-a71b-0ee3b26640d5" (UID: "73971904-e99b-4662-a71b-0ee3b26640d5"). InnerVolumeSpecName "kube-api-access-sdll6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:54:08.625658 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.625622    2584 scope.go:117] "RemoveContainer" containerID="99501850696cb9dd7b5210648c1cac6689b3e6366da91523833ca951abe27f30"
Sep 15 22:54:08.635402 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.635381    2584 scope.go:117] "RemoveContainer" containerID="1a1e2824716e61a5769e94aa7a14343f2d1e0163eff074d0f9428a75d8991d54"
Sep 15 22:54:08.655443 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.655416    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdll6\" (UniqueName: \"kubernetes.io/projected/73971904-e99b-4662-a71b-0ee3b26640d5-kube-api-access-sdll6\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:08.655523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.655444    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73971904-e99b-4662-a71b-0ee3b26640d5-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:08.711702 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.711634    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:08.771050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.771015    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54" event={"ID":"73971904-e99b-4662-a71b-0ee3b26640d5","Type":"ContainerDied","Data":"99501850696cb9dd7b5210648c1cac6689b3e6366da91523833ca951abe27f30"}
Sep 15 22:54:08.771189 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.771073    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"
Sep 15 22:54:08.771189 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.771074    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54" event={"ID":"73971904-e99b-4662-a71b-0ee3b26640d5","Type":"ContainerDied","Data":"3bb84f9e3a54da901d1005a1d2061fbaf0068c4eb884bf2d8119e5fcc7e386a9"}
Sep 15 22:54:08.811490 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.811456    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"]
Sep 15 22:54:08.815630 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.815608    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f2241f-cff6f9c55-c5h54"]
Sep 15 22:54:08.843739 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:08.843708    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:09.465774 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.465738    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:09.468001 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.467977    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss"]
Sep 15 22:54:09.472944 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.472913    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fe020d-29825205-hwrss"]
Sep 15 22:54:09.501971 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.501937    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"]
Sep 15 22:54:09.502224 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.502200    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j" podUID="22cf15a7-2d61-4507-a202-27a77939e4d2" containerName="payload-pre-processing" containerID="cri-o://1cc84c6a5ea369ef6be5e30e5a5e0b8c3ee178a00fa4963f129137e3cf3f3e35" gracePeriod=30
Sep 15 22:54:09.567356 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.567312    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:09.666674 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.666640    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:09.764525 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.764495    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:09.780299 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.780260    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"
Sep 15 22:54:09.781919 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.781892    2584 generic.go:356] "Generic (PLEG): container finished" podID="22cf15a7-2d61-4507-a202-27a77939e4d2" containerID="1cc84c6a5ea369ef6be5e30e5a5e0b8c3ee178a00fa4963f129137e3cf3f3e35" exitCode=0
Sep 15 22:54:09.782052 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.781962    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j" event={"ID":"22cf15a7-2d61-4507-a202-27a77939e4d2","Type":"ContainerDied","Data":"1cc84c6a5ea369ef6be5e30e5a5e0b8c3ee178a00fa4963f129137e3cf3f3e35"}
Sep 15 22:54:09.782052 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.782015    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j" event={"ID":"22cf15a7-2d61-4507-a202-27a77939e4d2","Type":"ContainerDied","Data":"53f6dc7081b9ecee561284ca4fc985d4452598e8bf195aa111195bb13defba38"}
Sep 15 22:54:09.782052 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.782044    2584 scope.go:117] "RemoveContainer" containerID="1cc84c6a5ea369ef6be5e30e5a5e0b8c3ee178a00fa4963f129137e3cf3f3e35"
Sep 15 22:54:09.797313 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.797200    2584 scope.go:117] "RemoveContainer" containerID="1cc84c6a5ea369ef6be5e30e5a5e0b8c3ee178a00fa4963f129137e3cf3f3e35"
Sep 15 22:54:09.797673 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:54:09.797642    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc84c6a5ea369ef6be5e30e5a5e0b8c3ee178a00fa4963f129137e3cf3f3e35\": container with ID starting with 1cc84c6a5ea369ef6be5e30e5a5e0b8c3ee178a00fa4963f129137e3cf3f3e35 not found: ID does not exist" containerID="1cc84c6a5ea369ef6be5e30e5a5e0b8c3ee178a00fa4963f129137e3cf3f3e35"
Sep 15 22:54:09.797773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.797681    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc84c6a5ea369ef6be5e30e5a5e0b8c3ee178a00fa4963f129137e3cf3f3e35"} err="failed to get container status \"1cc84c6a5ea369ef6be5e30e5a5e0b8c3ee178a00fa4963f129137e3cf3f3e35\": rpc error: code = NotFound desc = could not find container \"1cc84c6a5ea369ef6be5e30e5a5e0b8c3ee178a00fa4963f129137e3cf3f3e35\": container with ID starting with 1cc84c6a5ea369ef6be5e30e5a5e0b8c3ee178a00fa4963f129137e3cf3f3e35 not found: ID does not exist"
Sep 15 22:54:09.867519 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.867485    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cf15a7-2d61-4507-a202-27a77939e4d2-plugins-config-volume\") pod \"22cf15a7-2d61-4507-a202-27a77939e4d2\" (UID: \"22cf15a7-2d61-4507-a202-27a77939e4d2\") "
Sep 15 22:54:09.867671 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.867569    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvcvz\" (UniqueName: \"kubernetes.io/projected/22cf15a7-2d61-4507-a202-27a77939e4d2-kube-api-access-nvcvz\") pod \"22cf15a7-2d61-4507-a202-27a77939e4d2\" (UID: \"22cf15a7-2d61-4507-a202-27a77939e4d2\") "
Sep 15 22:54:09.867895 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.867870    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cf15a7-2d61-4507-a202-27a77939e4d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22cf15a7-2d61-4507-a202-27a77939e4d2" (UID: "22cf15a7-2d61-4507-a202-27a77939e4d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:54:09.869551 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.869531    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cf15a7-2d61-4507-a202-27a77939e4d2-kube-api-access-nvcvz" (OuterVolumeSpecName: "kube-api-access-nvcvz") pod "22cf15a7-2d61-4507-a202-27a77939e4d2" (UID: "22cf15a7-2d61-4507-a202-27a77939e4d2"). InnerVolumeSpecName "kube-api-access-nvcvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:54:09.900611 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.900586    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e9c13a-6949-435d-8df8-2813d66e7eb3" path="/var/lib/kubelet/pods/41e9c13a-6949-435d-8df8-2813d66e7eb3/volumes"
Sep 15 22:54:09.901038 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.901012    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73971904-e99b-4662-a71b-0ee3b26640d5" path="/var/lib/kubelet/pods/73971904-e99b-4662-a71b-0ee3b26640d5/volumes"
Sep 15 22:54:09.901293 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.901279    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ddf4114-aada-4e11-afe9-389c67dfa6e1" path="/var/lib/kubelet/pods/7ddf4114-aada-4e11-afe9-389c67dfa6e1/volumes"
Sep 15 22:54:09.922540 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.922511    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj"]
Sep 15 22:54:09.926360 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.926336    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fe020d-krqfj"]
Sep 15 22:54:09.927117 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.927091    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:09.968722 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.968698    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvcvz\" (UniqueName: \"kubernetes.io/projected/22cf15a7-2d61-4507-a202-27a77939e4d2-kube-api-access-nvcvz\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:09.968722 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:09.968722    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cf15a7-2d61-4507-a202-27a77939e4d2-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:10.026833 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:10.026752    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:10.296154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:10.296124    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:10.791101 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:10.791024    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"
Sep 15 22:54:10.811750 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:10.811715    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"]
Sep 15 22:54:10.819839 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:10.819812    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fe020d-5897bc68fd-k629j"]
Sep 15 22:54:10.983021 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:10.982988    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:11.766640 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:11.766595    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:11.899518 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:11.899477    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22cf15a7-2d61-4507-a202-27a77939e4d2" path="/var/lib/kubelet/pods/22cf15a7-2d61-4507-a202-27a77939e4d2/volumes"
Sep 15 22:54:11.900015 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:11.899992    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eade4bb3-6bb0-4783-ad02-8456bf107b8e" path="/var/lib/kubelet/pods/eade4bb3-6bb0-4783-ad02-8456bf107b8e/volumes"
Sep 15 22:54:12.578379 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:12.578339    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:13.448674 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:13.448635    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:14.132074 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:14.132041    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:14.952237 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:14.952207    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:15.032640 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.032604    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:15.205100 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.205009    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:15.524354 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.524248    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z"]
Sep 15 22:54:15.524696 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.524683    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eade4bb3-6bb0-4783-ad02-8456bf107b8e" containerName="revoke-keys"
Sep 15 22:54:15.524748 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.524697    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eade4bb3-6bb0-4783-ad02-8456bf107b8e" containerName="revoke-keys"
Sep 15 22:54:15.524748 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.524714    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e9c13a-6949-435d-8df8-2813d66e7eb3" containerName="revoke-keys"
Sep 15 22:54:15.524748 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.524720    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e9c13a-6949-435d-8df8-2813d66e7eb3" containerName="revoke-keys"
Sep 15 22:54:15.524748 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.524739    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22cf15a7-2d61-4507-a202-27a77939e4d2" containerName="payload-pre-processing"
Sep 15 22:54:15.524748 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.524744    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="22cf15a7-2d61-4507-a202-27a77939e4d2" containerName="payload-pre-processing"
Sep 15 22:54:15.524905 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.524750    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73971904-e99b-4662-a71b-0ee3b26640d5" containerName="payload-processing"
Sep 15 22:54:15.524905 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.524756    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="73971904-e99b-4662-a71b-0ee3b26640d5" containerName="payload-processing"
Sep 15 22:54:15.524905 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.524820    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eade4bb3-6bb0-4783-ad02-8456bf107b8e" containerName="revoke-keys"
Sep 15 22:54:15.524905 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.524830    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="22cf15a7-2d61-4507-a202-27a77939e4d2" containerName="payload-pre-processing"
Sep 15 22:54:15.524905 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.524836    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="73971904-e99b-4662-a71b-0ee3b26640d5" containerName="payload-processing"
Sep 15 22:54:15.524905 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.524846    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e9c13a-6949-435d-8df8-2813d66e7eb3" containerName="revoke-keys"
Sep 15 22:54:15.529279 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.529252    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z"
Sep 15 22:54:15.532157 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.532132    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:54:15.532291 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.532137    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:54:15.537066 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.536802    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z"]
Sep 15 22:54:15.622531 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.622498    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bebc29f-8813-4edf-baad-4ed0551cf164-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z\" (UID: \"7bebc29f-8813-4edf-baad-4ed0551cf164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z"
Sep 15 22:54:15.723707 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.723672    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bebc29f-8813-4edf-baad-4ed0551cf164-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z\" (UID: \"7bebc29f-8813-4edf-baad-4ed0551cf164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z"
Sep 15 22:54:15.724359 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.724340    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bebc29f-8813-4edf-baad-4ed0551cf164-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z\" (UID: \"7bebc29f-8813-4edf-baad-4ed0551cf164\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z"
Sep 15 22:54:15.841300 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:15.841237    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z"
Sep 15 22:54:16.115868 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:16.115840    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z"]
Sep 15 22:54:16.118202 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:54:16.118166    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bebc29f_8813_4edf_baad_4ed0551cf164.slice/crio-4060b159b837ac97d69474df5fed3a4394c26c8b344a776a3a5703cf06404f5d WatchSource:0}: Error finding container 4060b159b837ac97d69474df5fed3a4394c26c8b344a776a3a5703cf06404f5d: Status 404 returned error can't find the container with id 4060b159b837ac97d69474df5fed3a4394c26c8b344a776a3a5703cf06404f5d
Sep 15 22:54:16.371829 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:16.371736    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:16.479663 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:16.479633    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:16.602646 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:16.602609    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:16.663731 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:16.663657    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"]
Sep 15 22:54:16.663901 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:16.663874    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy" containerID="cri-o://01ab1a33bb2f3a46cd0861275be2253a0be216842816af8ba669de826b2d9a12" gracePeriod=30
Sep 15 22:54:16.739844 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:16.739762    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:16.830310 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:16.830278    2584 generic.go:356] "Generic (PLEG): container finished" podID="7bebc29f-8813-4edf-baad-4ed0551cf164" containerID="2540a5f4f4a411e4592b68a866021c1e0f7140adc36025e283a97882389368ec" exitCode=0
Sep 15 22:54:16.830440 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:16.830363    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z" event={"ID":"7bebc29f-8813-4edf-baad-4ed0551cf164","Type":"ContainerDied","Data":"2540a5f4f4a411e4592b68a866021c1e0f7140adc36025e283a97882389368ec"}
Sep 15 22:54:16.830440 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:16.830420    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z" event={"ID":"7bebc29f-8813-4edf-baad-4ed0551cf164","Type":"ContainerStarted","Data":"4060b159b837ac97d69474df5fed3a4394c26c8b344a776a3a5703cf06404f5d"}
Sep 15 22:54:17.373886 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:17.373847    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:17.868717 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:17.868666    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh" podUID="8dafd07c-2342-4799-803a-061c6fe6488d" containerName="main" containerID="cri-o://00992e64db88b3c6091bfa3a46a7aeae491979c38fb8c75aed3fa3e0fdd9ede1" gracePeriod=45
Sep 15 22:54:17.880718 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:17.880682    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6" podUID="524703bc-7a31-4028-b790-543f88a6dcd9" containerName="main" containerID="cri-o://28bbfbc95745484411e62a5d1398b72c453c1855f4bd983747462d076f029eed" gracePeriod=45
Sep 15 22:54:18.008864 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.008839    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z"
Sep 15 22:54:18.046933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.046261    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bebc29f-8813-4edf-baad-4ed0551cf164-maas-api-service-ca\") pod \"7bebc29f-8813-4edf-baad-4ed0551cf164\" (UID: \"7bebc29f-8813-4edf-baad-4ed0551cf164\") "
Sep 15 22:54:18.047333 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.047284    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bebc29f-8813-4edf-baad-4ed0551cf164-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7bebc29f-8813-4edf-baad-4ed0551cf164" (UID: "7bebc29f-8813-4edf-baad-4ed0551cf164"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:54:18.095235 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.095140    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:18.148225 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.148192    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bebc29f-8813-4edf-baad-4ed0551cf164-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:18.164371 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.164346    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:54:18.170410 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.170389    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:54:18.248848 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.248818    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-model-cache\") pod \"524703bc-7a31-4028-b790-543f88a6dcd9\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") "
Sep 15 22:54:18.249036 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.248868    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/524703bc-7a31-4028-b790-543f88a6dcd9-tls-certs\") pod \"524703bc-7a31-4028-b790-543f88a6dcd9\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") "
Sep 15 22:54:18.249176 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.249146    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-model-cache" (OuterVolumeSpecName: "model-cache") pod "524703bc-7a31-4028-b790-543f88a6dcd9" (UID: "524703bc-7a31-4028-b790-543f88a6dcd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:18.249296 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.249235    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-home\") pod \"524703bc-7a31-4028-b790-543f88a6dcd9\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") "
Sep 15 22:54:18.249374 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.249315    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-home\") pod \"8dafd07c-2342-4799-803a-061c6fe6488d\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") "
Sep 15 22:54:18.249374 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.249353    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twngd\" (UniqueName: \"kubernetes.io/projected/524703bc-7a31-4028-b790-543f88a6dcd9-kube-api-access-twngd\") pod \"524703bc-7a31-4028-b790-543f88a6dcd9\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") "
Sep 15 22:54:18.249491 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.249387    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-tmp-dir\") pod \"8dafd07c-2342-4799-803a-061c6fe6488d\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") "
Sep 15 22:54:18.249491 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.249417    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-model-cache\") pod \"8dafd07c-2342-4799-803a-061c6fe6488d\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") "
Sep 15 22:54:18.249610 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.249541    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bps2h\" (UniqueName: \"kubernetes.io/projected/8dafd07c-2342-4799-803a-061c6fe6488d-kube-api-access-bps2h\") pod \"8dafd07c-2342-4799-803a-061c6fe6488d\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") "
Sep 15 22:54:18.249610 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.249583    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dafd07c-2342-4799-803a-061c6fe6488d-tls-certs\") pod \"8dafd07c-2342-4799-803a-061c6fe6488d\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") "
Sep 15 22:54:18.249718 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.249631    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-dshm\") pod \"8dafd07c-2342-4799-803a-061c6fe6488d\" (UID: \"8dafd07c-2342-4799-803a-061c6fe6488d\") "
Sep 15 22:54:18.249718 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.249678    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-dshm\") pod \"524703bc-7a31-4028-b790-543f88a6dcd9\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") "
Sep 15 22:54:18.249718 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.249709    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-tmp-dir\") pod \"524703bc-7a31-4028-b790-543f88a6dcd9\" (UID: \"524703bc-7a31-4028-b790-543f88a6dcd9\") "
Sep 15 22:54:18.250163 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.250139    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:18.250163 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.250138    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-home" (OuterVolumeSpecName: "home") pod "524703bc-7a31-4028-b790-543f88a6dcd9" (UID: "524703bc-7a31-4028-b790-543f88a6dcd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:18.250467 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.250440    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8dafd07c-2342-4799-803a-061c6fe6488d" (UID: "8dafd07c-2342-4799-803a-061c6fe6488d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:18.250700 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.250666    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8dafd07c-2342-4799-803a-061c6fe6488d" (UID: "8dafd07c-2342-4799-803a-061c6fe6488d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:18.250944 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.250902    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-home" (OuterVolumeSpecName: "home") pod "8dafd07c-2342-4799-803a-061c6fe6488d" (UID: "8dafd07c-2342-4799-803a-061c6fe6488d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:18.251363 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.251329    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "524703bc-7a31-4028-b790-543f88a6dcd9" (UID: "524703bc-7a31-4028-b790-543f88a6dcd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:18.252714 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.252672    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-dshm" (OuterVolumeSpecName: "dshm") pod "524703bc-7a31-4028-b790-543f88a6dcd9" (UID: "524703bc-7a31-4028-b790-543f88a6dcd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:18.252933 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.252911    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/524703bc-7a31-4028-b790-543f88a6dcd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "524703bc-7a31-4028-b790-543f88a6dcd9" (UID: "524703bc-7a31-4028-b790-543f88a6dcd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:54:18.253583 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.253559    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-dshm" (OuterVolumeSpecName: "dshm") pod "8dafd07c-2342-4799-803a-061c6fe6488d" (UID: "8dafd07c-2342-4799-803a-061c6fe6488d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:18.253740 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.253709    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dafd07c-2342-4799-803a-061c6fe6488d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dafd07c-2342-4799-803a-061c6fe6488d" (UID: "8dafd07c-2342-4799-803a-061c6fe6488d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:54:18.254432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.254405    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/524703bc-7a31-4028-b790-543f88a6dcd9-kube-api-access-twngd" (OuterVolumeSpecName: "kube-api-access-twngd") pod "524703bc-7a31-4028-b790-543f88a6dcd9" (UID: "524703bc-7a31-4028-b790-543f88a6dcd9"). InnerVolumeSpecName "kube-api-access-twngd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:54:18.254586 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.254554    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dafd07c-2342-4799-803a-061c6fe6488d-kube-api-access-bps2h" (OuterVolumeSpecName: "kube-api-access-bps2h") pod "8dafd07c-2342-4799-803a-061c6fe6488d" (UID: "8dafd07c-2342-4799-803a-061c6fe6488d"). InnerVolumeSpecName "kube-api-access-bps2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:54:18.350982 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.350902    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:18.350982 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.350930    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:18.350982 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.350958    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twngd\" (UniqueName: \"kubernetes.io/projected/524703bc-7a31-4028-b790-543f88a6dcd9-kube-api-access-twngd\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:18.350982 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.350968    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:18.350982 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.350977    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:18.350982 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.350985    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bps2h\" (UniqueName: \"kubernetes.io/projected/8dafd07c-2342-4799-803a-061c6fe6488d-kube-api-access-bps2h\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:18.351291 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.350993    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dafd07c-2342-4799-803a-061c6fe6488d-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:18.351291 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.351001    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dafd07c-2342-4799-803a-061c6fe6488d-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:18.351291 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.351007    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:18.351291 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.351014    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/524703bc-7a31-4028-b790-543f88a6dcd9-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:18.351291 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.351021    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/524703bc-7a31-4028-b790-543f88a6dcd9-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:18.842421 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.842393    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z"
Sep 15 22:54:18.842830 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.842432    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z" event={"ID":"7bebc29f-8813-4edf-baad-4ed0551cf164","Type":"ContainerDied","Data":"4060b159b837ac97d69474df5fed3a4394c26c8b344a776a3a5703cf06404f5d"}
Sep 15 22:54:18.842830 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.842523    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4060b159b837ac97d69474df5fed3a4394c26c8b344a776a3a5703cf06404f5d"
Sep 15 22:54:18.843795 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.843771    2584 generic.go:356] "Generic (PLEG): container finished" podID="524703bc-7a31-4028-b790-543f88a6dcd9" containerID="28bbfbc95745484411e62a5d1398b72c453c1855f4bd983747462d076f029eed" exitCode=0
Sep 15 22:54:18.843899 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.843821    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"
Sep 15 22:54:18.843899 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.843849    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6" event={"ID":"524703bc-7a31-4028-b790-543f88a6dcd9","Type":"ContainerDied","Data":"28bbfbc95745484411e62a5d1398b72c453c1855f4bd983747462d076f029eed"}
Sep 15 22:54:18.843899 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.843896    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6" event={"ID":"524703bc-7a31-4028-b790-543f88a6dcd9","Type":"ContainerDied","Data":"21ba4cec8d06dc3da8217e7dbd9a59bdc3021f2c7671e0fe9a405c01a17ae92e"}
Sep 15 22:54:18.844020 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.843922    2584 scope.go:117] "RemoveContainer" containerID="28bbfbc95745484411e62a5d1398b72c453c1855f4bd983747462d076f029eed"
Sep 15 22:54:18.845330 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.845284    2584 generic.go:356] "Generic (PLEG): container finished" podID="8dafd07c-2342-4799-803a-061c6fe6488d" containerID="00992e64db88b3c6091bfa3a46a7aeae491979c38fb8c75aed3fa3e0fdd9ede1" exitCode=0
Sep 15 22:54:18.845412 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.845336    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"
Sep 15 22:54:18.845412 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.845371    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh" event={"ID":"8dafd07c-2342-4799-803a-061c6fe6488d","Type":"ContainerDied","Data":"00992e64db88b3c6091bfa3a46a7aeae491979c38fb8c75aed3fa3e0fdd9ede1"}
Sep 15 22:54:18.845521 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.845411    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh" event={"ID":"8dafd07c-2342-4799-803a-061c6fe6488d","Type":"ContainerDied","Data":"9ff2c79bd35a831c0d68cf19e3f43672fd1d51c84f12b5ed3244469939a38945"}
Sep 15 22:54:18.855146 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.855128    2584 scope.go:117] "RemoveContainer" containerID="28bbfbc95745484411e62a5d1398b72c453c1855f4bd983747462d076f029eed"
Sep 15 22:54:18.855422 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:54:18.855403    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bbfbc95745484411e62a5d1398b72c453c1855f4bd983747462d076f029eed\": container with ID starting with 28bbfbc95745484411e62a5d1398b72c453c1855f4bd983747462d076f029eed not found: ID does not exist" containerID="28bbfbc95745484411e62a5d1398b72c453c1855f4bd983747462d076f029eed"
Sep 15 22:54:18.855492 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.855428    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bbfbc95745484411e62a5d1398b72c453c1855f4bd983747462d076f029eed"} err="failed to get container status \"28bbfbc95745484411e62a5d1398b72c453c1855f4bd983747462d076f029eed\": rpc error: code = NotFound desc = could not find container \"28bbfbc95745484411e62a5d1398b72c453c1855f4bd983747462d076f029eed\": container with ID starting with 28bbfbc95745484411e62a5d1398b72c453c1855f4bd983747462d076f029eed not found: ID does not exist"
Sep 15 22:54:18.855492 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.855445    2584 scope.go:117] "RemoveContainer" containerID="00992e64db88b3c6091bfa3a46a7aeae491979c38fb8c75aed3fa3e0fdd9ede1"
Sep 15 22:54:18.863698 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.863682    2584 scope.go:117] "RemoveContainer" containerID="00992e64db88b3c6091bfa3a46a7aeae491979c38fb8c75aed3fa3e0fdd9ede1"
Sep 15 22:54:18.863941 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:54:18.863925    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00992e64db88b3c6091bfa3a46a7aeae491979c38fb8c75aed3fa3e0fdd9ede1\": container with ID starting with 00992e64db88b3c6091bfa3a46a7aeae491979c38fb8c75aed3fa3e0fdd9ede1 not found: ID does not exist" containerID="00992e64db88b3c6091bfa3a46a7aeae491979c38fb8c75aed3fa3e0fdd9ede1"
Sep 15 22:54:18.863987 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.863947    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00992e64db88b3c6091bfa3a46a7aeae491979c38fb8c75aed3fa3e0fdd9ede1"} err="failed to get container status \"00992e64db88b3c6091bfa3a46a7aeae491979c38fb8c75aed3fa3e0fdd9ede1\": rpc error: code = NotFound desc = could not find container \"00992e64db88b3c6091bfa3a46a7aeae491979c38fb8c75aed3fa3e0fdd9ede1\": container with ID starting with 00992e64db88b3c6091bfa3a46a7aeae491979c38fb8c75aed3fa3e0fdd9ede1 not found: ID does not exist"
Sep 15 22:54:18.870784 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.870763    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"]
Sep 15 22:54:18.876178 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.876155    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fe020d/distinct-w0-fe020d-kserve-b64c474d8-4x7w6"]
Sep 15 22:54:18.886921 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.886882    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"]
Sep 15 22:54:18.890674 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:18.890654    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fe020d/distinct-2-w0-fe020d-kserve-f89687c95-vmnkh"]
Sep 15 22:54:19.062318 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:19.062281    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:19.743116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:19.743084    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:19.898974 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:19.898936    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="524703bc-7a31-4028-b790-543f88a6dcd9" path="/var/lib/kubelet/pods/524703bc-7a31-4028-b790-543f88a6dcd9/volumes"
Sep 15 22:54:19.899502 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:19.899482    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dafd07c-2342-4799-803a-061c6fe6488d" path="/var/lib/kubelet/pods/8dafd07c-2342-4799-803a-061c6fe6488d/volumes"
Sep 15 22:54:21.866897 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.866870    2584 generic.go:356] "Generic (PLEG): container finished" podID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerID="01ab1a33bb2f3a46cd0861275be2253a0be216842816af8ba669de826b2d9a12" exitCode=0
Sep 15 22:54:21.867281 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.866947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" event={"ID":"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca","Type":"ContainerDied","Data":"01ab1a33bb2f3a46cd0861275be2253a0be216842816af8ba669de826b2d9a12"}
Sep 15 22:54:21.867281 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.867021    2584 scope.go:117] "RemoveContainer" containerID="c48c3ba34829c989333d521e47efca8c4f0f28a144ff85afcd020b29a8d25e95"
Sep 15 22:54:21.927571 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.927547    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:54:21.988796 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.988714    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-credential-socket\") pod \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") "
Sep 15 22:54:21.988796 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.988759    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-workload-socket\") pod \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") "
Sep 15 22:54:21.988796 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.988785    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istiod-ca-cert\") pod \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") "
Sep 15 22:54:21.989069 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.988806    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb9hx\" (UniqueName: \"kubernetes.io/projected/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-kube-api-access-nb9hx\") pod \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") "
Sep 15 22:54:21.989069 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.988830    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-podinfo\") pod \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") "
Sep 15 22:54:21.989069 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.988846    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-token\") pod \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") "
Sep 15 22:54:21.989069 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.988898    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-data\") pod \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") "
Sep 15 22:54:21.989069 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.989012    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" (UID: "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:21.989069 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.989067    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-envoy\") pod \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") "
Sep 15 22:54:21.989413 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.989162    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" (UID: "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:21.989413 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.989189    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-workload-certs\") pod \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\" (UID: \"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca\") "
Sep 15 22:54:21.989413 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.989209    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" (UID: "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:54:21.989413 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.989367    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" (UID: "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:21.989618 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.989549    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" (UID: "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:21.989966 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.989945    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istiod-ca-cert\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:21.990050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.989970    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-data\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:21.990050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.989984    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-workload-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:21.990050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.989995    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-credential-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:21.990050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.990008    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-workload-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:21.991353 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.991302    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-kube-api-access-nb9hx" (OuterVolumeSpecName: "kube-api-access-nb9hx") pod "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" (UID: "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca"). InnerVolumeSpecName "kube-api-access-nb9hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:54:21.991353 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.991316    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" (UID: "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:21.991518 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.991500    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" (UID: "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 22:54:21.991880 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:21.991852    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" (UID: "35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:54:22.091547 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:22.091513    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb9hx\" (UniqueName: \"kubernetes.io/projected/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-kube-api-access-nb9hx\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:22.091547 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:22.091545    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-podinfo\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:22.091733 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:22.091555    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-token\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:22.091733 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:22.091564    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca-istio-envoy\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:22.874044 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:22.874011    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"
Sep 15 22:54:22.874474 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:22.874049    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f" event={"ID":"35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca","Type":"ContainerDied","Data":"3147ee17c7fb05a633d55b1f998b8f4537c171588bc2bc91ce2b7863473e686b"}
Sep 15 22:54:22.874474 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:22.874110    2584 scope.go:117] "RemoveContainer" containerID="01ab1a33bb2f3a46cd0861275be2253a0be216842816af8ba669de826b2d9a12"
Sep 15 22:54:22.897807 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:22.897784    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"]
Sep 15 22:54:22.901696 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:22.901671    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe020d-openshift-default-65cdd9995d-zhb4f"]
Sep 15 22:54:23.899449 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:23.899415    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" path="/var/lib/kubelet/pods/35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca/volumes"
Sep 15 22:54:26.054191 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:26.054147    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:26.124165 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:26.124122    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:26.192406 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:26.192362    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:26.245315 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:26.245260    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z"]
Sep 15 22:54:26.248382 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:26.248358    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f70a9-fsh9z"]
Sep 15 22:54:26.262865 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:26.262834    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:26.576828 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:26.576788    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:26.678516 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:26.678485    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:27.394532 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:27.394497    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:27.899583 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:27.899550    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bebc29f-8813-4edf-baad-4ed0551cf164" path="/var/lib/kubelet/pods/7bebc29f-8813-4edf-baad-4ed0551cf164/volumes"
Sep 15 22:54:28.178582 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:28.178496    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:28.921955 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:28.921899    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:32.595465 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:32.595426    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:32.696066 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:32.696033    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:32.845026 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:32.844990    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:33.041494 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.041410    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:33.150789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.150752    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj"]
Sep 15 22:54:33.151154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151142    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dafd07c-2342-4799-803a-061c6fe6488d" containerName="main"
Sep 15 22:54:33.151200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151157    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dafd07c-2342-4799-803a-061c6fe6488d" containerName="main"
Sep 15 22:54:33.151200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151165    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy"
Sep 15 22:54:33.151200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151172    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy"
Sep 15 22:54:33.151200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151179    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bebc29f-8813-4edf-baad-4ed0551cf164" containerName="revoke-keys"
Sep 15 22:54:33.151200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151185    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bebc29f-8813-4edf-baad-4ed0551cf164" containerName="revoke-keys"
Sep 15 22:54:33.151200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151196    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy"
Sep 15 22:54:33.151200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151203    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy"
Sep 15 22:54:33.151432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151218    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="524703bc-7a31-4028-b790-543f88a6dcd9" containerName="main"
Sep 15 22:54:33.151432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151223    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="524703bc-7a31-4028-b790-543f88a6dcd9" containerName="main"
Sep 15 22:54:33.151432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151315    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy"
Sep 15 22:54:33.151432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151325    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="524703bc-7a31-4028-b790-543f88a6dcd9" containerName="main"
Sep 15 22:54:33.151432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151333    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dafd07c-2342-4799-803a-061c6fe6488d" containerName="main"
Sep 15 22:54:33.151432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.151339    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bebc29f-8813-4edf-baad-4ed0551cf164" containerName="revoke-keys"
Sep 15 22:54:33.155895 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.155877    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj"
Sep 15 22:54:33.158788 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.158767    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:54:33.159880 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.159857    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 22:54:33.163051 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.163026    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj"]
Sep 15 22:54:33.296836 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.296803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a281743-bf28-4558-82dc-5b29ab50c0f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj\" (UID: \"5a281743-bf28-4558-82dc-5b29ab50c0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj"
Sep 15 22:54:33.398181 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.398147    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a281743-bf28-4558-82dc-5b29ab50c0f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj\" (UID: \"5a281743-bf28-4558-82dc-5b29ab50c0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj"
Sep 15 22:54:33.398839 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.398819    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a281743-bf28-4558-82dc-5b29ab50c0f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj\" (UID: \"5a281743-bf28-4558-82dc-5b29ab50c0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj"
Sep 15 22:54:33.418789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.418757    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:33.467866 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.467840    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj"
Sep 15 22:54:33.743582 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.743556    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj"]
Sep 15 22:54:33.745462 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:54:33.745436    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a281743_bf28_4558_82dc_5b29ab50c0f4.slice/crio-5e0f512efb7427490bbc183486fef1a3dd521af972054f850c67e7e8836d7340 WatchSource:0}: Error finding container 5e0f512efb7427490bbc183486fef1a3dd521af972054f850c67e7e8836d7340: Status 404 returned error can't find the container with id 5e0f512efb7427490bbc183486fef1a3dd521af972054f850c67e7e8836d7340
Sep 15 22:54:33.938244 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:33.938210    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj" event={"ID":"5a281743-bf28-4558-82dc-5b29ab50c0f4","Type":"ContainerStarted","Data":"5e0f512efb7427490bbc183486fef1a3dd521af972054f850c67e7e8836d7340"}
Sep 15 22:54:34.157548 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:34.157506    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:34.945523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:34.945486    2584 generic.go:356] "Generic (PLEG): container finished" podID="5a281743-bf28-4558-82dc-5b29ab50c0f4" containerID="19eb4411609b6b0af77a0f88de53116f990071237bf5b974ebdd5ac57113bd34" exitCode=0
Sep 15 22:54:34.945902 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:34.945547    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj" event={"ID":"5a281743-bf28-4558-82dc-5b29ab50c0f4","Type":"ContainerDied","Data":"19eb4411609b6b0af77a0f88de53116f990071237bf5b974ebdd5ac57113bd34"}
Sep 15 22:54:34.967561 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:34.967528    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:35.750137 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:35.750094    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:36.088410 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:36.088384    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj"
Sep 15 22:54:36.229196 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:36.229165    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a281743-bf28-4558-82dc-5b29ab50c0f4-maas-api-service-ca\") pod \"5a281743-bf28-4558-82dc-5b29ab50c0f4\" (UID: \"5a281743-bf28-4558-82dc-5b29ab50c0f4\") "
Sep 15 22:54:36.229550 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:36.229527    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a281743-bf28-4558-82dc-5b29ab50c0f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a281743-bf28-4558-82dc-5b29ab50c0f4" (UID: "5a281743-bf28-4558-82dc-5b29ab50c0f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:54:36.330487 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:36.330446    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a281743-bf28-4558-82dc-5b29ab50c0f4-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:36.457136 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:36.457100    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:36.958252 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:36.958217    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj" event={"ID":"5a281743-bf28-4558-82dc-5b29ab50c0f4","Type":"ContainerDied","Data":"5e0f512efb7427490bbc183486fef1a3dd521af972054f850c67e7e8836d7340"}
Sep 15 22:54:36.958252 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:36.958225    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj"
Sep 15 22:54:36.958498 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:36.958285    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e0f512efb7427490bbc183486fef1a3dd521af972054f850c67e7e8836d7340"
Sep 15 22:54:38.017559 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:38.017520    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"]
Sep 15 22:54:38.023584 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:38.023553    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"]
Sep 15 22:54:38.036142 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:38.036111    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"]
Sep 15 22:54:38.107685 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:38.107640    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:39.461310 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:39.461277    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:54:39.871162 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:39.871127    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:39.928111 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:39.928071    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:40.322011 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:40.321974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:45.916903 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:45.916857    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:47.385831 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:47.385789    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:48.836455 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:48.836403    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:48.846932 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:48.846902    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw"]
Sep 15 22:54:48.849056 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:48.849031    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-03f258-29825205-m2vrw"]
Sep 15 22:54:48.877533 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:48.877508    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"]
Sep 15 22:54:48.877745 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:48.877720    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd" podUID="aa43a50f-3be7-464c-836b-5554361e62ea" containerName="payload-processing" containerID="cri-o://e1f3ee3da179a5dc2b63242c6b7de84407b2fb7246064d556060d2961c8fd3f5" gracePeriod=30
Sep 15 22:54:48.885349 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:48.885325    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"]
Sep 15 22:54:48.885810 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:48.885519    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd" podUID="5a2d55aa-1749-434c-947f-9cdbc6a76375" containerName="payload-pre-processing" containerID="cri-o://e8ce1ccd58ee1dbdd35d2012fae78517bed816b7a9a01c8c2077741a227fbc27" gracePeriod=30
Sep 15 22:54:48.924580 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:48.924551    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:48.982803 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:48.982770    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:49.031558 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.031526    2584 generic.go:356] "Generic (PLEG): container finished" podID="aa43a50f-3be7-464c-836b-5554361e62ea" containerID="e1f3ee3da179a5dc2b63242c6b7de84407b2fb7246064d556060d2961c8fd3f5" exitCode=0
Sep 15 22:54:49.031804 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.031710    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd" event={"ID":"aa43a50f-3be7-464c-836b-5554361e62ea","Type":"ContainerDied","Data":"e1f3ee3da179a5dc2b63242c6b7de84407b2fb7246064d556060d2961c8fd3f5"}
Sep 15 22:54:49.033545 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.033514    2584 generic.go:356] "Generic (PLEG): container finished" podID="5a2d55aa-1749-434c-947f-9cdbc6a76375" containerID="e8ce1ccd58ee1dbdd35d2012fae78517bed816b7a9a01c8c2077741a227fbc27" exitCode=0
Sep 15 22:54:49.033655 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.033569    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd" event={"ID":"5a2d55aa-1749-434c-947f-9cdbc6a76375","Type":"ContainerDied","Data":"e8ce1ccd58ee1dbdd35d2012fae78517bed816b7a9a01c8c2077741a227fbc27"}
Sep 15 22:54:49.060256 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.060227    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj"]
Sep 15 22:54:49.062489 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.062418    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:49.065361 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.065332    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-03f258-kz7nj"]
Sep 15 22:54:49.127907 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.127875    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:49.173953 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.173929    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"
Sep 15 22:54:49.177080 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.177061    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"
Sep 15 22:54:49.242225 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.242198    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h45sm\" (UniqueName: \"kubernetes.io/projected/5a2d55aa-1749-434c-947f-9cdbc6a76375-kube-api-access-h45sm\") pod \"5a2d55aa-1749-434c-947f-9cdbc6a76375\" (UID: \"5a2d55aa-1749-434c-947f-9cdbc6a76375\") "
Sep 15 22:54:49.242397 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.242366    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnhgr\" (UniqueName: \"kubernetes.io/projected/aa43a50f-3be7-464c-836b-5554361e62ea-kube-api-access-gnhgr\") pod \"aa43a50f-3be7-464c-836b-5554361e62ea\" (UID: \"aa43a50f-3be7-464c-836b-5554361e62ea\") "
Sep 15 22:54:49.242397 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.242390    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa43a50f-3be7-464c-836b-5554361e62ea-plugins-config-volume\") pod \"aa43a50f-3be7-464c-836b-5554361e62ea\" (UID: \"aa43a50f-3be7-464c-836b-5554361e62ea\") "
Sep 15 22:54:49.242481 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.242422    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2d55aa-1749-434c-947f-9cdbc6a76375-plugins-config-volume\") pod \"5a2d55aa-1749-434c-947f-9cdbc6a76375\" (UID: \"5a2d55aa-1749-434c-947f-9cdbc6a76375\") "
Sep 15 22:54:49.242836 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.242807    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa43a50f-3be7-464c-836b-5554361e62ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa43a50f-3be7-464c-836b-5554361e62ea" (UID: "aa43a50f-3be7-464c-836b-5554361e62ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:54:49.242959 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.242861    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2d55aa-1749-434c-947f-9cdbc6a76375-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a2d55aa-1749-434c-947f-9cdbc6a76375" (UID: "5a2d55aa-1749-434c-947f-9cdbc6a76375"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:54:49.244491 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.244463    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa43a50f-3be7-464c-836b-5554361e62ea-kube-api-access-gnhgr" (OuterVolumeSpecName: "kube-api-access-gnhgr") pod "aa43a50f-3be7-464c-836b-5554361e62ea" (UID: "aa43a50f-3be7-464c-836b-5554361e62ea"). InnerVolumeSpecName "kube-api-access-gnhgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:54:49.244491 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.244478    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2d55aa-1749-434c-947f-9cdbc6a76375-kube-api-access-h45sm" (OuterVolumeSpecName: "kube-api-access-h45sm") pod "5a2d55aa-1749-434c-947f-9cdbc6a76375" (UID: "5a2d55aa-1749-434c-947f-9cdbc6a76375"). InnerVolumeSpecName "kube-api-access-h45sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:54:49.344318 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.344223    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h45sm\" (UniqueName: \"kubernetes.io/projected/5a2d55aa-1749-434c-947f-9cdbc6a76375-kube-api-access-h45sm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:49.344318 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.344251    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnhgr\" (UniqueName: \"kubernetes.io/projected/aa43a50f-3be7-464c-836b-5554361e62ea-kube-api-access-gnhgr\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:49.344318 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.344262    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa43a50f-3be7-464c-836b-5554361e62ea-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:49.344318 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.344299    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2d55aa-1749-434c-947f-9cdbc6a76375-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:49.415525 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.415491    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:49.900289 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.900196    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a281743-bf28-4558-82dc-5b29ab50c0f4" path="/var/lib/kubelet/pods/5a281743-bf28-4558-82dc-5b29ab50c0f4/volumes"
Sep 15 22:54:49.900607 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:49.900518    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ce505db-5217-4724-9496-5c34d460b9cc" path="/var/lib/kubelet/pods/5ce505db-5217-4724-9496-5c34d460b9cc/volumes"
Sep 15 22:54:50.041566 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:50.041531    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"
Sep 15 22:54:50.041732 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:50.041563    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd" event={"ID":"5a2d55aa-1749-434c-947f-9cdbc6a76375","Type":"ContainerDied","Data":"6039eb60a6ffe489c05004ab517695b65398db4e62ad619393e9d68ac3c27d4b"}
Sep 15 22:54:50.041732 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:50.041622    2584 scope.go:117] "RemoveContainer" containerID="e8ce1ccd58ee1dbdd35d2012fae78517bed816b7a9a01c8c2077741a227fbc27"
Sep 15 22:54:50.043223 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:50.043200    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd" event={"ID":"aa43a50f-3be7-464c-836b-5554361e62ea","Type":"ContainerDied","Data":"d756cc57f137d1b49ca80f57ca5c8d6d2558a49eff1f86eefc1cd25a9cfafc99"}
Sep 15 22:54:50.043425 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:50.043209    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"
Sep 15 22:54:50.051324 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:50.051292    2584 scope.go:117] "RemoveContainer" containerID="e1f3ee3da179a5dc2b63242c6b7de84407b2fb7246064d556060d2961c8fd3f5"
Sep 15 22:54:50.061177 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:50.061154    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"]
Sep 15 22:54:50.064275 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:50.064242    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-03f258-b6c875758-q78xd"]
Sep 15 22:54:50.076424 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:50.076405    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"]
Sep 15 22:54:50.082311 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:50.082287    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:50.083151 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:50.083129    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-03f258-c9bc9b9f5-7qnjd"]
Sep 15 22:54:50.787317 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:50.787284    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:51.816320 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:51.816288    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:51.899562 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:51.899531    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a2d55aa-1749-434c-947f-9cdbc6a76375" path="/var/lib/kubelet/pods/5a2d55aa-1749-434c-947f-9cdbc6a76375/volumes"
Sep 15 22:54:51.899908 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:51.899894    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa43a50f-3be7-464c-836b-5554361e62ea" path="/var/lib/kubelet/pods/aa43a50f-3be7-464c-836b-5554361e62ea/volumes"
Sep 15 22:54:53.037551 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.037482    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4" podUID="20834e61-d5e9-4fb3-8fc9-1939228873c0" containerName="main" containerID="cri-o://6f01a878ac2a629168b6c81e602ea836ca7c475f6d75c43b931df30f5f4832fa" gracePeriod=45
Sep 15 22:54:53.056728 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.056639    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6" podUID="3829049a-52f5-44b9-8ea2-de5558590665" containerName="main" containerID="cri-o://827a28b966e9235cf4f9d330c46fbb943aa1051a26a073e1b79d5349a5405b1f" gracePeriod=45
Sep 15 22:54:53.056819 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.056640    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t" podUID="01901ba0-7ab7-40ce-a716-512106879cfe" containerName="main" containerID="cri-o://b96b9b9e52d36e39d6c158c3a1923bc78f56dd3a5051f1b3fa2c94cd4554a9e4" gracePeriod=45
Sep 15 22:54:53.347807 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.347779    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:54:53.352108 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.352079    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:54:53.355763 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.355741    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:54:53.383722 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.383696    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-model-cache\") pod \"3829049a-52f5-44b9-8ea2-de5558590665\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") "
Sep 15 22:54:53.383878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.383767    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-dshm\") pod \"01901ba0-7ab7-40ce-a716-512106879cfe\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") "
Sep 15 22:54:53.383878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.383795    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvpjd\" (UniqueName: \"kubernetes.io/projected/20834e61-d5e9-4fb3-8fc9-1939228873c0-kube-api-access-gvpjd\") pod \"20834e61-d5e9-4fb3-8fc9-1939228873c0\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") "
Sep 15 22:54:53.383878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.383837    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-model-cache\") pod \"20834e61-d5e9-4fb3-8fc9-1939228873c0\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") "
Sep 15 22:54:53.383878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.383867    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-home\") pod \"01901ba0-7ab7-40ce-a716-512106879cfe\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") "
Sep 15 22:54:53.384116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.383896    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-tmp-dir\") pod \"01901ba0-7ab7-40ce-a716-512106879cfe\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") "
Sep 15 22:54:53.384116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.383937    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3829049a-52f5-44b9-8ea2-de5558590665-tls-certs\") pod \"3829049a-52f5-44b9-8ea2-de5558590665\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") "
Sep 15 22:54:53.384116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.383966    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01901ba0-7ab7-40ce-a716-512106879cfe-tls-certs\") pod \"01901ba0-7ab7-40ce-a716-512106879cfe\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") "
Sep 15 22:54:53.384116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.383990    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-dshm\") pod \"3829049a-52f5-44b9-8ea2-de5558590665\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") "
Sep 15 22:54:53.384116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.383995    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-model-cache" (OuterVolumeSpecName: "model-cache") pod "3829049a-52f5-44b9-8ea2-de5558590665" (UID: "3829049a-52f5-44b9-8ea2-de5558590665"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:53.384116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.384015    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-tmp-dir\") pod \"3829049a-52f5-44b9-8ea2-de5558590665\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") "
Sep 15 22:54:53.384116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.384048    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krpdt\" (UniqueName: \"kubernetes.io/projected/3829049a-52f5-44b9-8ea2-de5558590665-kube-api-access-krpdt\") pod \"3829049a-52f5-44b9-8ea2-de5558590665\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") "
Sep 15 22:54:53.384116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.384074    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-tmp-dir\") pod \"20834e61-d5e9-4fb3-8fc9-1939228873c0\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") "
Sep 15 22:54:53.384116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.384104    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-dshm\") pod \"20834e61-d5e9-4fb3-8fc9-1939228873c0\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") "
Sep 15 22:54:53.384596 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.384130    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-model-cache\") pod \"01901ba0-7ab7-40ce-a716-512106879cfe\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") "
Sep 15 22:54:53.384596 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.384184    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01901ba0-7ab7-40ce-a716-512106879cfe" (UID: "01901ba0-7ab7-40ce-a716-512106879cfe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:53.384596 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.384565    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20834e61-d5e9-4fb3-8fc9-1939228873c0-tls-certs\") pod \"20834e61-d5e9-4fb3-8fc9-1939228873c0\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") "
Sep 15 22:54:53.384750 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.384621    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "20834e61-d5e9-4fb3-8fc9-1939228873c0" (UID: "20834e61-d5e9-4fb3-8fc9-1939228873c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:53.384750 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.384636    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-home\") pod \"20834e61-d5e9-4fb3-8fc9-1939228873c0\" (UID: \"20834e61-d5e9-4fb3-8fc9-1939228873c0\") "
Sep 15 22:54:53.384750 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.384728    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-home\") pod \"3829049a-52f5-44b9-8ea2-de5558590665\" (UID: \"3829049a-52f5-44b9-8ea2-de5558590665\") "
Sep 15 22:54:53.384902 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.384810    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62pjf\" (UniqueName: \"kubernetes.io/projected/01901ba0-7ab7-40ce-a716-512106879cfe-kube-api-access-62pjf\") pod \"01901ba0-7ab7-40ce-a716-512106879cfe\" (UID: \"01901ba0-7ab7-40ce-a716-512106879cfe\") "
Sep 15 22:54:53.384902 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.384836    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-home" (OuterVolumeSpecName: "home") pod "20834e61-d5e9-4fb3-8fc9-1939228873c0" (UID: "20834e61-d5e9-4fb3-8fc9-1939228873c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:53.388154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.385065    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3829049a-52f5-44b9-8ea2-de5558590665" (UID: "3829049a-52f5-44b9-8ea2-de5558590665"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:53.388154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.385172    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-home" (OuterVolumeSpecName: "home") pod "01901ba0-7ab7-40ce-a716-512106879cfe" (UID: "01901ba0-7ab7-40ce-a716-512106879cfe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:53.388154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.385249    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.388154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.385285    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.388154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.385299    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.388154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.385310    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.388154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.385321    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.388154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.385365    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-home" (OuterVolumeSpecName: "home") pod "3829049a-52f5-44b9-8ea2-de5558590665" (UID: "3829049a-52f5-44b9-8ea2-de5558590665"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:53.388154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.385648    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-model-cache" (OuterVolumeSpecName: "model-cache") pod "01901ba0-7ab7-40ce-a716-512106879cfe" (UID: "01901ba0-7ab7-40ce-a716-512106879cfe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:53.388154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.385859    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20834e61-d5e9-4fb3-8fc9-1939228873c0" (UID: "20834e61-d5e9-4fb3-8fc9-1939228873c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:53.391047 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.388477    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20834e61-d5e9-4fb3-8fc9-1939228873c0-kube-api-access-gvpjd" (OuterVolumeSpecName: "kube-api-access-gvpjd") pod "20834e61-d5e9-4fb3-8fc9-1939228873c0" (UID: "20834e61-d5e9-4fb3-8fc9-1939228873c0"). InnerVolumeSpecName "kube-api-access-gvpjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:54:53.391047 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.389831    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3829049a-52f5-44b9-8ea2-de5558590665-kube-api-access-krpdt" (OuterVolumeSpecName: "kube-api-access-krpdt") pod "3829049a-52f5-44b9-8ea2-de5558590665" (UID: "3829049a-52f5-44b9-8ea2-de5558590665"). InnerVolumeSpecName "kube-api-access-krpdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:54:53.391047 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.390083    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01901ba0-7ab7-40ce-a716-512106879cfe-kube-api-access-62pjf" (OuterVolumeSpecName: "kube-api-access-62pjf") pod "01901ba0-7ab7-40ce-a716-512106879cfe" (UID: "01901ba0-7ab7-40ce-a716-512106879cfe"). InnerVolumeSpecName "kube-api-access-62pjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:54:53.391212 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.391190    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20834e61-d5e9-4fb3-8fc9-1939228873c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20834e61-d5e9-4fb3-8fc9-1939228873c0" (UID: "20834e61-d5e9-4fb3-8fc9-1939228873c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:54:53.391445 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.391190    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-dshm" (OuterVolumeSpecName: "dshm") pod "20834e61-d5e9-4fb3-8fc9-1939228873c0" (UID: "20834e61-d5e9-4fb3-8fc9-1939228873c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:53.391445 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.391313    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01901ba0-7ab7-40ce-a716-512106879cfe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01901ba0-7ab7-40ce-a716-512106879cfe" (UID: "01901ba0-7ab7-40ce-a716-512106879cfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:54:53.391445 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.391368    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-dshm" (OuterVolumeSpecName: "dshm") pod "3829049a-52f5-44b9-8ea2-de5558590665" (UID: "3829049a-52f5-44b9-8ea2-de5558590665"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:53.391445 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.391386    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-dshm" (OuterVolumeSpecName: "dshm") pod "01901ba0-7ab7-40ce-a716-512106879cfe" (UID: "01901ba0-7ab7-40ce-a716-512106879cfe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:54:53.392571 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.392549    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3829049a-52f5-44b9-8ea2-de5558590665-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3829049a-52f5-44b9-8ea2-de5558590665" (UID: "3829049a-52f5-44b9-8ea2-de5558590665"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:54:53.486146 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.486118    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.486146 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.486144    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62pjf\" (UniqueName: \"kubernetes.io/projected/01901ba0-7ab7-40ce-a716-512106879cfe-kube-api-access-62pjf\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.486146 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.486155    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.486376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.486164    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvpjd\" (UniqueName: \"kubernetes.io/projected/20834e61-d5e9-4fb3-8fc9-1939228873c0-kube-api-access-gvpjd\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.486376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.486173    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.486376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.486182    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3829049a-52f5-44b9-8ea2-de5558590665-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.486376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.486190    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01901ba0-7ab7-40ce-a716-512106879cfe-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.486376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.486197    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3829049a-52f5-44b9-8ea2-de5558590665-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.486376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.486205    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krpdt\" (UniqueName: \"kubernetes.io/projected/3829049a-52f5-44b9-8ea2-de5558590665-kube-api-access-krpdt\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.486376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.486212    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.486376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.486220    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20834e61-d5e9-4fb3-8fc9-1939228873c0-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.486376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.486227    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01901ba0-7ab7-40ce-a716-512106879cfe-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:53.486376 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:53.486234    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20834e61-d5e9-4fb3-8fc9-1939228873c0-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:54:54.069567 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.069530    2584 generic.go:356] "Generic (PLEG): container finished" podID="3829049a-52f5-44b9-8ea2-de5558590665" containerID="827a28b966e9235cf4f9d330c46fbb943aa1051a26a073e1b79d5349a5405b1f" exitCode=0
Sep 15 22:54:54.069962 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.069580    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"
Sep 15 22:54:54.069962 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.069625    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6" event={"ID":"3829049a-52f5-44b9-8ea2-de5558590665","Type":"ContainerDied","Data":"827a28b966e9235cf4f9d330c46fbb943aa1051a26a073e1b79d5349a5405b1f"}
Sep 15 22:54:54.069962 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.069677    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6" event={"ID":"3829049a-52f5-44b9-8ea2-de5558590665","Type":"ContainerDied","Data":"e01adf8cd1695eb2efe5cd75a4d339ee586dda1dc89d12e744841d5b5358727b"}
Sep 15 22:54:54.069962 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.069702    2584 scope.go:117] "RemoveContainer" containerID="827a28b966e9235cf4f9d330c46fbb943aa1051a26a073e1b79d5349a5405b1f"
Sep 15 22:54:54.071124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.071104    2584 generic.go:356] "Generic (PLEG): container finished" podID="20834e61-d5e9-4fb3-8fc9-1939228873c0" containerID="6f01a878ac2a629168b6c81e602ea836ca7c475f6d75c43b931df30f5f4832fa" exitCode=0
Sep 15 22:54:54.071197 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.071142    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"
Sep 15 22:54:54.071197 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.071180    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4" event={"ID":"20834e61-d5e9-4fb3-8fc9-1939228873c0","Type":"ContainerDied","Data":"6f01a878ac2a629168b6c81e602ea836ca7c475f6d75c43b931df30f5f4832fa"}
Sep 15 22:54:54.071434 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.071219    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4" event={"ID":"20834e61-d5e9-4fb3-8fc9-1939228873c0","Type":"ContainerDied","Data":"a7dcf51aefbf51ecb189ed5a42e1920f1342dab38058da24c297a9a6d798d7c3"}
Sep 15 22:54:54.073039 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.072976    2584 generic.go:356] "Generic (PLEG): container finished" podID="01901ba0-7ab7-40ce-a716-512106879cfe" containerID="b96b9b9e52d36e39d6c158c3a1923bc78f56dd3a5051f1b3fa2c94cd4554a9e4" exitCode=0
Sep 15 22:54:54.073284 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.073081    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"
Sep 15 22:54:54.073284 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.073100    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t" event={"ID":"01901ba0-7ab7-40ce-a716-512106879cfe","Type":"ContainerDied","Data":"b96b9b9e52d36e39d6c158c3a1923bc78f56dd3a5051f1b3fa2c94cd4554a9e4"}
Sep 15 22:54:54.073284 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.073254    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t" event={"ID":"01901ba0-7ab7-40ce-a716-512106879cfe","Type":"ContainerDied","Data":"b8cbe8048e8a252677f877ed90e5854f37a7d305398644dbfbc0935dd6cae8c2"}
Sep 15 22:54:54.079825 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.079802    2584 scope.go:117] "RemoveContainer" containerID="827a28b966e9235cf4f9d330c46fbb943aa1051a26a073e1b79d5349a5405b1f"
Sep 15 22:54:54.080080 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:54:54.080063    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827a28b966e9235cf4f9d330c46fbb943aa1051a26a073e1b79d5349a5405b1f\": container with ID starting with 827a28b966e9235cf4f9d330c46fbb943aa1051a26a073e1b79d5349a5405b1f not found: ID does not exist" containerID="827a28b966e9235cf4f9d330c46fbb943aa1051a26a073e1b79d5349a5405b1f"
Sep 15 22:54:54.080125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.080086    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827a28b966e9235cf4f9d330c46fbb943aa1051a26a073e1b79d5349a5405b1f"} err="failed to get container status \"827a28b966e9235cf4f9d330c46fbb943aa1051a26a073e1b79d5349a5405b1f\": rpc error: code = NotFound desc = could not find container \"827a28b966e9235cf4f9d330c46fbb943aa1051a26a073e1b79d5349a5405b1f\": container with ID starting with 827a28b966e9235cf4f9d330c46fbb943aa1051a26a073e1b79d5349a5405b1f not found: ID does not exist"
Sep 15 22:54:54.080125 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.080100    2584 scope.go:117] "RemoveContainer" containerID="6f01a878ac2a629168b6c81e602ea836ca7c475f6d75c43b931df30f5f4832fa"
Sep 15 22:54:54.086992 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.086967    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"]
Sep 15 22:54:54.090420 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.090396    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/distinct-2-w4-03f258-kserve-76d584fdf6-7cqk4"]
Sep 15 22:54:54.091755 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.091741    2584 scope.go:117] "RemoveContainer" containerID="6f01a878ac2a629168b6c81e602ea836ca7c475f6d75c43b931df30f5f4832fa"
Sep 15 22:54:54.092000 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:54:54.091981    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f01a878ac2a629168b6c81e602ea836ca7c475f6d75c43b931df30f5f4832fa\": container with ID starting with 6f01a878ac2a629168b6c81e602ea836ca7c475f6d75c43b931df30f5f4832fa not found: ID does not exist" containerID="6f01a878ac2a629168b6c81e602ea836ca7c475f6d75c43b931df30f5f4832fa"
Sep 15 22:54:54.092050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.092007    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f01a878ac2a629168b6c81e602ea836ca7c475f6d75c43b931df30f5f4832fa"} err="failed to get container status \"6f01a878ac2a629168b6c81e602ea836ca7c475f6d75c43b931df30f5f4832fa\": rpc error: code = NotFound desc = could not find container \"6f01a878ac2a629168b6c81e602ea836ca7c475f6d75c43b931df30f5f4832fa\": container with ID starting with 6f01a878ac2a629168b6c81e602ea836ca7c475f6d75c43b931df30f5f4832fa not found: ID does not exist"
Sep 15 22:54:54.092050 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.092024    2584 scope.go:117] "RemoveContainer" containerID="b96b9b9e52d36e39d6c158c3a1923bc78f56dd3a5051f1b3fa2c94cd4554a9e4"
Sep 15 22:54:54.099702 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.099684    2584 scope.go:117] "RemoveContainer" containerID="b96b9b9e52d36e39d6c158c3a1923bc78f56dd3a5051f1b3fa2c94cd4554a9e4"
Sep 15 22:54:54.099970 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:54:54.099952    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96b9b9e52d36e39d6c158c3a1923bc78f56dd3a5051f1b3fa2c94cd4554a9e4\": container with ID starting with b96b9b9e52d36e39d6c158c3a1923bc78f56dd3a5051f1b3fa2c94cd4554a9e4 not found: ID does not exist" containerID="b96b9b9e52d36e39d6c158c3a1923bc78f56dd3a5051f1b3fa2c94cd4554a9e4"
Sep 15 22:54:54.100024 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.099976    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96b9b9e52d36e39d6c158c3a1923bc78f56dd3a5051f1b3fa2c94cd4554a9e4"} err="failed to get container status \"b96b9b9e52d36e39d6c158c3a1923bc78f56dd3a5051f1b3fa2c94cd4554a9e4\": rpc error: code = NotFound desc = could not find container \"b96b9b9e52d36e39d6c158c3a1923bc78f56dd3a5051f1b3fa2c94cd4554a9e4\": container with ID starting with b96b9b9e52d36e39d6c158c3a1923bc78f56dd3a5051f1b3fa2c94cd4554a9e4 not found: ID does not exist"
Sep 15 22:54:54.100068 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.100048    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"]
Sep 15 22:54:54.103307 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.103286    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/distinct-w4-03f258-kserve-76585b58ff-brfg6"]
Sep 15 22:54:54.117045 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.117026    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"]
Sep 15 22:54:54.120415 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:54.120395    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-03f258/premium-w4-03f258-kserve-bdd55895b-kkh8t"]
Sep 15 22:54:55.878886 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:55.878850    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:55.898533 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:55.898503    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01901ba0-7ab7-40ce-a716-512106879cfe" path="/var/lib/kubelet/pods/01901ba0-7ab7-40ce-a716-512106879cfe/volumes"
Sep 15 22:54:55.898927 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:55.898914    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20834e61-d5e9-4fb3-8fc9-1939228873c0" path="/var/lib/kubelet/pods/20834e61-d5e9-4fb3-8fc9-1939228873c0/volumes"
Sep 15 22:54:55.899262 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:55.899250    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3829049a-52f5-44b9-8ea2-de5558590665" path="/var/lib/kubelet/pods/3829049a-52f5-44b9-8ea2-de5558590665/volumes"
Sep 15 22:54:55.940312 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:55.940281    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:54:56.218780 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:54:56.218699    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:55:02.366073 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:02.366034    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:55:08.672013 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:08.671980    2584 scope.go:117] "RemoveContainer" containerID="de4c0042f09ad4b3e786b67f8f942d9bbfe0815503cee7209715a20b96506689"
Sep 15 22:55:08.680405 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:08.680383    2584 scope.go:117] "RemoveContainer" containerID="e1b4be3cb23a830e64c686cce5686c8f9c45e6d8622e8d7cd7b77b8902213bef"
Sep 15 22:55:08.687765 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:08.687747    2584 scope.go:117] "RemoveContainer" containerID="81d6ee0cca89ccddd67f487f440ecfba4440ea85cf0f619f2920462cd90e143d"
Sep 15 22:55:14.201589 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:14.201528    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:55:14.301842 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:14.301804    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:55:16.084331 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:16.084297    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:55:17.358808 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:17.358774    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:55:21.159434 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:21.159392    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:55:21.820752 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:21.820705    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:55:30.559510 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:30.559424    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:55:37.633455 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633414    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7654-315725"]
Sep 15 22:55:37.633852 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633835    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20834e61-d5e9-4fb3-8fc9-1939228873c0" containerName="main"
Sep 15 22:55:37.633852 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633848    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="20834e61-d5e9-4fb3-8fc9-1939228873c0" containerName="main"
Sep 15 22:55:37.633923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633859    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa43a50f-3be7-464c-836b-5554361e62ea" containerName="payload-processing"
Sep 15 22:55:37.633923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633866    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa43a50f-3be7-464c-836b-5554361e62ea" containerName="payload-processing"
Sep 15 22:55:37.633923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633877    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01901ba0-7ab7-40ce-a716-512106879cfe" containerName="main"
Sep 15 22:55:37.633923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633883    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="01901ba0-7ab7-40ce-a716-512106879cfe" containerName="main"
Sep 15 22:55:37.633923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633894    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a281743-bf28-4558-82dc-5b29ab50c0f4" containerName="revoke-keys"
Sep 15 22:55:37.633923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633899    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a281743-bf28-4558-82dc-5b29ab50c0f4" containerName="revoke-keys"
Sep 15 22:55:37.633923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633907    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3829049a-52f5-44b9-8ea2-de5558590665" containerName="main"
Sep 15 22:55:37.633923 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633912    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3829049a-52f5-44b9-8ea2-de5558590665" containerName="main"
Sep 15 22:55:37.634165 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633926    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2d55aa-1749-434c-947f-9cdbc6a76375" containerName="payload-pre-processing"
Sep 15 22:55:37.634165 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633931    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2d55aa-1749-434c-947f-9cdbc6a76375" containerName="payload-pre-processing"
Sep 15 22:55:37.634165 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633985    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3829049a-52f5-44b9-8ea2-de5558590665" containerName="main"
Sep 15 22:55:37.634165 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633991    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a2d55aa-1749-434c-947f-9cdbc6a76375" containerName="payload-pre-processing"
Sep 15 22:55:37.634165 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.633998    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="01901ba0-7ab7-40ce-a716-512106879cfe" containerName="main"
Sep 15 22:55:37.634165 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.634006    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a281743-bf28-4558-82dc-5b29ab50c0f4" containerName="revoke-keys"
Sep 15 22:55:37.634165 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.634017    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="20834e61-d5e9-4fb3-8fc9-1939228873c0" containerName="main"
Sep 15 22:55:37.634165 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.634026    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa43a50f-3be7-464c-836b-5554361e62ea" containerName="payload-processing"
Sep 15 22:55:37.634165 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.634037    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="35e10f06-2ee4-40bb-b2cc-71cb3e0c55ca" containerName="istio-proxy"
Sep 15 22:55:37.637310 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.637294    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-315725"
Sep 15 22:55:37.640148 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.640125    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4fdg9\""
Sep 15 22:55:37.642289 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.642249    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7654-315725"]
Sep 15 22:55:37.682385 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.682357    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbd4k\" (UniqueName: \"kubernetes.io/projected/699ce367-60c1-480d-8aef-0bd14f5f00ef-kube-api-access-kbd4k\") pod \"test-curl-7654-315725\" (UID: \"699ce367-60c1-480d-8aef-0bd14f5f00ef\") " pod="opendatahub/test-curl-7654-315725"
Sep 15 22:55:37.783605 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.783575    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbd4k\" (UniqueName: \"kubernetes.io/projected/699ce367-60c1-480d-8aef-0bd14f5f00ef-kube-api-access-kbd4k\") pod \"test-curl-7654-315725\" (UID: \"699ce367-60c1-480d-8aef-0bd14f5f00ef\") " pod="opendatahub/test-curl-7654-315725"
Sep 15 22:55:37.792319 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.792296    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbd4k\" (UniqueName: \"kubernetes.io/projected/699ce367-60c1-480d-8aef-0bd14f5f00ef-kube-api-access-kbd4k\") pod \"test-curl-7654-315725\" (UID: \"699ce367-60c1-480d-8aef-0bd14f5f00ef\") " pod="opendatahub/test-curl-7654-315725"
Sep 15 22:55:37.947781 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:37.947696    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-315725"
Sep 15 22:55:38.216521 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:38.216496    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7654-315725"]
Sep 15 22:55:38.218816 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:55:38.218786    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod699ce367_60c1_480d_8aef_0bd14f5f00ef.slice/crio-4a6c85d4a468e1863d8bc0bd2268eee331f8845a628a530d4e52023358dc1bb6 WatchSource:0}: Error finding container 4a6c85d4a468e1863d8bc0bd2268eee331f8845a628a530d4e52023358dc1bb6: Status 404 returned error can't find the container with id 4a6c85d4a468e1863d8bc0bd2268eee331f8845a628a530d4e52023358dc1bb6
Sep 15 22:55:38.220985 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:38.220965    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 22:55:38.318839 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:38.318800    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7654-315725" event={"ID":"699ce367-60c1-480d-8aef-0bd14f5f00ef","Type":"ContainerStarted","Data":"4a6c85d4a468e1863d8bc0bd2268eee331f8845a628a530d4e52023358dc1bb6"}
Sep 15 22:55:40.331880 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:40.331844    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7654-315725" event={"ID":"699ce367-60c1-480d-8aef-0bd14f5f00ef","Type":"ContainerStarted","Data":"8f048ad50b2c18956d67632ad1ecdce5807852e940ce503d70e77daf0f7022cf"}
Sep 15 22:55:40.347045 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:40.347003    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7654-315725" podStartSLOduration=1.450402754 podStartE2EDuration="3.346990699s" totalImagesPullingTime="1.896587945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:55:40.34659209 +0000 UTC m=+1712.965406998" watchObservedRunningTime="2026-09-15 22:55:40.346990699 +0000 UTC m=+1712.965805606"
Sep 15 22:55:40.930832 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:40.930793    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7654-315725"]
Sep 15 22:55:40.934573 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:40.934541    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7654-315725"]
Sep 15 22:55:41.337328 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:55:41.337296    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7654-315725" container="test-curl-7654-315725"
Sep 15 22:55:42.268279 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:42.268227    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:55:42.343517 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:42.343481    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7654-315725" podUID="699ce367-60c1-480d-8aef-0bd14f5f00ef" containerName="test-curl-7654-315725" containerID="cri-o://8f048ad50b2c18956d67632ad1ecdce5807852e940ce503d70e77daf0f7022cf" gracePeriod=2
Sep 15 22:55:42.466496 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:42.466466    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:55:43.238491 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:43.238459    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"]
Sep 15 22:55:43.260788 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:43.260757    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:55:44.517130 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:44.517106    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-315725"
Sep 15 22:55:44.647167 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:44.647096    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbd4k\" (UniqueName: \"kubernetes.io/projected/699ce367-60c1-480d-8aef-0bd14f5f00ef-kube-api-access-kbd4k\") pod \"699ce367-60c1-480d-8aef-0bd14f5f00ef\" (UID: \"699ce367-60c1-480d-8aef-0bd14f5f00ef\") "
Sep 15 22:55:44.649155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:44.649127    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699ce367-60c1-480d-8aef-0bd14f5f00ef-kube-api-access-kbd4k" (OuterVolumeSpecName: "kube-api-access-kbd4k") pod "699ce367-60c1-480d-8aef-0bd14f5f00ef" (UID: "699ce367-60c1-480d-8aef-0bd14f5f00ef"). InnerVolumeSpecName "kube-api-access-kbd4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:55:44.747683 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:44.747654    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbd4k\" (UniqueName: \"kubernetes.io/projected/699ce367-60c1-480d-8aef-0bd14f5f00ef-kube-api-access-kbd4k\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:55:45.359246 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:45.359213    2584 generic.go:356] "Generic (PLEG): container finished" podID="699ce367-60c1-480d-8aef-0bd14f5f00ef" containerID="8f048ad50b2c18956d67632ad1ecdce5807852e940ce503d70e77daf0f7022cf" exitCode=137
Sep 15 22:55:45.359437 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:45.359251    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-315725"
Sep 15 22:55:45.359437 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:45.359320    2584 scope.go:117] "RemoveContainer" containerID="8f048ad50b2c18956d67632ad1ecdce5807852e940ce503d70e77daf0f7022cf"
Sep 15 22:55:45.368460 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:45.368444    2584 scope.go:117] "RemoveContainer" containerID="8f048ad50b2c18956d67632ad1ecdce5807852e940ce503d70e77daf0f7022cf"
Sep 15 22:55:45.368723 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:55:45.368703    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f048ad50b2c18956d67632ad1ecdce5807852e940ce503d70e77daf0f7022cf\": container with ID starting with 8f048ad50b2c18956d67632ad1ecdce5807852e940ce503d70e77daf0f7022cf not found: ID does not exist" containerID="8f048ad50b2c18956d67632ad1ecdce5807852e940ce503d70e77daf0f7022cf"
Sep 15 22:55:45.368777 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:45.368730    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f048ad50b2c18956d67632ad1ecdce5807852e940ce503d70e77daf0f7022cf"} err="failed to get container status \"8f048ad50b2c18956d67632ad1ecdce5807852e940ce503d70e77daf0f7022cf\": rpc error: code = NotFound desc = could not find container \"8f048ad50b2c18956d67632ad1ecdce5807852e940ce503d70e77daf0f7022cf\": container with ID starting with 8f048ad50b2c18956d67632ad1ecdce5807852e940ce503d70e77daf0f7022cf not found: ID does not exist"
Sep 15 22:55:45.370037 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:55:45.370013    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7654-315725\" is forbidden: User \"system:node:ip-10-0-138-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-116.ec2.internal' and this object" podUID="699ce367-60c1-480d-8aef-0bd14f5f00ef" pod="opendatahub/test-curl-7654-315725"
Sep 15 22:55:45.898775 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:45.898739    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="699ce367-60c1-480d-8aef-0bd14f5f00ef" path="/var/lib/kubelet/pods/699ce367-60c1-480d-8aef-0bd14f5f00ef/volumes"
Sep 15 22:55:51.373712 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:51.373673    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:55:52.977507 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:52.977474    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"]
Sep 15 22:55:52.977846 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:52.977838    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="699ce367-60c1-480d-8aef-0bd14f5f00ef" containerName="test-curl-7654-315725"
Sep 15 22:55:52.977898 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:52.977850    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="699ce367-60c1-480d-8aef-0bd14f5f00ef" containerName="test-curl-7654-315725"
Sep 15 22:55:52.977939 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:52.977931    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="699ce367-60c1-480d-8aef-0bd14f5f00ef" containerName="test-curl-7654-315725"
Sep 15 22:55:52.982546 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:52.982528    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:52.985481 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:52.985445    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0016d4\"/\"kube-root-ca.crt\""
Sep 15 22:55:52.986692 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:52.986670    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0016d4\"/\"sub-test-model-0016d4-kserve-self-signed-certs\""
Sep 15 22:55:52.986818 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:52.986677    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0016d4\"/\"openshift-service-ca.crt\""
Sep 15 22:55:52.987189 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:52.987168    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0016d4\"/\"default-dockercfg-nhjlf\""
Sep 15 22:55:52.989441 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:52.989417    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"]
Sep 15 22:55:52.991059 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:52.991020    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:55:53.121873 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.121837    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd6a3323-22ee-414a-b5f7-b780e25bb16c-tls-certs\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.122059 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.121888    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-dshm\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.122059 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.121980    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-model-cache\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.122059 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.122002    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cb4v\" (UniqueName: \"kubernetes.io/projected/dd6a3323-22ee-414a-b5f7-b780e25bb16c-kube-api-access-4cb4v\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.122226 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.122064    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-tmp-dir\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.122226 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.122112    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-home\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.222875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.222839    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-model-cache\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.222875 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.222878    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cb4v\" (UniqueName: \"kubernetes.io/projected/dd6a3323-22ee-414a-b5f7-b780e25bb16c-kube-api-access-4cb4v\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.223070 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.222899    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-tmp-dir\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.223070 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.222931    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-home\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.223070 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.222968    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd6a3323-22ee-414a-b5f7-b780e25bb16c-tls-certs\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.223070 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.223013    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-dshm\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.223336 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.223315    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-model-cache\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.223389 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.223358    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-tmp-dir\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.223389 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.223368    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-home\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.225238 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.225218    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-dshm\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.225450 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.225423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd6a3323-22ee-414a-b5f7-b780e25bb16c-tls-certs\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.230817 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.230766    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cb4v\" (UniqueName: \"kubernetes.io/projected/dd6a3323-22ee-414a-b5f7-b780e25bb16c-kube-api-access-4cb4v\") pod \"sub-test-model-0016d4-kserve-c857666cc-p25sk\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") " pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.293597 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.293569    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:55:53.569584 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:53.569552    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"]
Sep 15 22:55:53.571627 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:55:53.571602    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd6a3323_22ee_414a_b5f7_b780e25bb16c.slice/crio-d1074682a1a1809dc1e07d550f649a41bc27d730057705f39bdd36161ef38803 WatchSource:0}: Error finding container d1074682a1a1809dc1e07d550f649a41bc27d730057705f39bdd36161ef38803: Status 404 returned error can't find the container with id d1074682a1a1809dc1e07d550f649a41bc27d730057705f39bdd36161ef38803
Sep 15 22:55:54.408664 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:54.408626    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk" event={"ID":"dd6a3323-22ee-414a-b5f7-b780e25bb16c","Type":"ContainerStarted","Data":"34339c5dad77c35afddf8faef7e6a73a5bd56bc26b1bfd8ee999d45427f78291"}
Sep 15 22:55:54.409135 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:54.408673    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk" event={"ID":"dd6a3323-22ee-414a-b5f7-b780e25bb16c","Type":"ContainerStarted","Data":"d1074682a1a1809dc1e07d550f649a41bc27d730057705f39bdd36161ef38803"}
Sep 15 22:55:54.425721 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:54.425613    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk" podStartSLOduration=2.425597292 podStartE2EDuration="2.425597292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:55:54.424965503 +0000 UTC m=+1727.043780411" watchObservedRunningTime="2026-09-15 22:55:54.425597292 +0000 UTC m=+1727.044412201"
Sep 15 22:55:58.257799 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.257734    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm" podUID="1458f10f-2bfc-448f-b5c0-383aff19e70b" containerName="main" containerID="cri-o://c3ca9f862a8175b5856d2d4ce2af5b1d329aab4e32072a3efecf00b9c76d577c" gracePeriod=45
Sep 15 22:55:58.432373 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.432339    2584 generic.go:356] "Generic (PLEG): container finished" podID="1458f10f-2bfc-448f-b5c0-383aff19e70b" containerID="c3ca9f862a8175b5856d2d4ce2af5b1d329aab4e32072a3efecf00b9c76d577c" exitCode=0
Sep 15 22:55:58.432504 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.432409    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm" event={"ID":"1458f10f-2bfc-448f-b5c0-383aff19e70b","Type":"ContainerDied","Data":"c3ca9f862a8175b5856d2d4ce2af5b1d329aab4e32072a3efecf00b9c76d577c"}
Sep 15 22:55:58.517028 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.516979    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:55:58.669257 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.669226    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-home\") pod \"1458f10f-2bfc-448f-b5c0-383aff19e70b\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") "
Sep 15 22:55:58.669447 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.669304    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-dshm\") pod \"1458f10f-2bfc-448f-b5c0-383aff19e70b\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") "
Sep 15 22:55:58.669447 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.669333    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-model-cache\") pod \"1458f10f-2bfc-448f-b5c0-383aff19e70b\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") "
Sep 15 22:55:58.669447 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.669365    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1458f10f-2bfc-448f-b5c0-383aff19e70b-tls-certs\") pod \"1458f10f-2bfc-448f-b5c0-383aff19e70b\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") "
Sep 15 22:55:58.669447 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.669402    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-tmp-dir\") pod \"1458f10f-2bfc-448f-b5c0-383aff19e70b\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") "
Sep 15 22:55:58.669664 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.669468    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzmtw\" (UniqueName: \"kubernetes.io/projected/1458f10f-2bfc-448f-b5c0-383aff19e70b-kube-api-access-rzmtw\") pod \"1458f10f-2bfc-448f-b5c0-383aff19e70b\" (UID: \"1458f10f-2bfc-448f-b5c0-383aff19e70b\") "
Sep 15 22:55:58.669664 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.669480    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-home" (OuterVolumeSpecName: "home") pod "1458f10f-2bfc-448f-b5c0-383aff19e70b" (UID: "1458f10f-2bfc-448f-b5c0-383aff19e70b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:55:58.669664 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.669590    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1458f10f-2bfc-448f-b5c0-383aff19e70b" (UID: "1458f10f-2bfc-448f-b5c0-383aff19e70b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:55:58.669822 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.669674    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1458f10f-2bfc-448f-b5c0-383aff19e70b" (UID: "1458f10f-2bfc-448f-b5c0-383aff19e70b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:55:58.669822 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.669808    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:55:58.669919 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.669822    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:55:58.669919 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.669833    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:55:58.671610 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.671580    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-dshm" (OuterVolumeSpecName: "dshm") pod "1458f10f-2bfc-448f-b5c0-383aff19e70b" (UID: "1458f10f-2bfc-448f-b5c0-383aff19e70b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:55:58.671713 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.671647    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1458f10f-2bfc-448f-b5c0-383aff19e70b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1458f10f-2bfc-448f-b5c0-383aff19e70b" (UID: "1458f10f-2bfc-448f-b5c0-383aff19e70b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:55:58.671713 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.671693    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1458f10f-2bfc-448f-b5c0-383aff19e70b-kube-api-access-rzmtw" (OuterVolumeSpecName: "kube-api-access-rzmtw") pod "1458f10f-2bfc-448f-b5c0-383aff19e70b" (UID: "1458f10f-2bfc-448f-b5c0-383aff19e70b"). InnerVolumeSpecName "kube-api-access-rzmtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:55:58.770784 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.770720    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1458f10f-2bfc-448f-b5c0-383aff19e70b-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:55:58.770784 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.770742    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1458f10f-2bfc-448f-b5c0-383aff19e70b-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:55:58.770784 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:58.770753    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzmtw\" (UniqueName: \"kubernetes.io/projected/1458f10f-2bfc-448f-b5c0-383aff19e70b-kube-api-access-rzmtw\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:55:59.438699 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:59.438670    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"
Sep 15 22:55:59.439153 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:59.438727    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm" event={"ID":"1458f10f-2bfc-448f-b5c0-383aff19e70b","Type":"ContainerDied","Data":"b174c0473fd1250d555baceb486178b480eb9922784c72dd4e9ae44675006d24"}
Sep 15 22:55:59.439153 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:59.438788    2584 scope.go:117] "RemoveContainer" containerID="c3ca9f862a8175b5856d2d4ce2af5b1d329aab4e32072a3efecf00b9c76d577c"
Sep 15 22:55:59.460370 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:59.460341    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"]
Sep 15 22:55:59.466225 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:59.466203    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/auth-test-model-92482b-kserve-5b8865f85f-v8krm"]
Sep 15 22:55:59.898810 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:55:59.898776    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1458f10f-2bfc-448f-b5c0-383aff19e70b" path="/var/lib/kubelet/pods/1458f10f-2bfc-448f-b5c0-383aff19e70b/volumes"
Sep 15 22:56:01.060078 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:01.060035    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:56:03.294071 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:03.294039    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:56:03.294544 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:03.294342    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:56:03.306392 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:03.306368    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:56:03.473990 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:03.473960    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:56:08.753999 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:08.753965    2584 scope.go:117] "RemoveContainer" containerID="5e34cd033b2f10e39b8b605bdaccc3c805c20baf7728e230340b30883b3cd95c"
Sep 15 22:56:22.061533 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.061495    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"]
Sep 15 22:56:22.062079 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.062062    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1458f10f-2bfc-448f-b5c0-383aff19e70b" containerName="main"
Sep 15 22:56:22.062128 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.062082    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1458f10f-2bfc-448f-b5c0-383aff19e70b" containerName="main"
Sep 15 22:56:22.062206 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.062195    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1458f10f-2bfc-448f-b5c0-383aff19e70b" containerName="main"
Sep 15 22:56:22.070450 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.070422    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.073479 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.073452    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-92482b\"/\"kube-root-ca.crt\""
Sep 15 22:56:22.073631 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.073513    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-92482b\"/\"sub-test-model-92482b-kserve-self-signed-certs\""
Sep 15 22:56:22.075363 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.075336    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"]
Sep 15 22:56:22.075713 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.075693    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-92482b\"/\"default-dockercfg-nckqq\""
Sep 15 22:56:22.075898 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.075706    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-92482b\"/\"openshift-service-ca.crt\""
Sep 15 22:56:22.094992 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.094962    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:56:22.180058 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.180026    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887fce08-5e8e-427c-974d-7b2225077411-tls-certs\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.180204 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.180076    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-home\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.180204 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.180102    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-model-cache\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.180204 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.180163    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-dshm\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.180204 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.180202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-tmp-dir\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.180474 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.180230    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwffq\" (UniqueName: \"kubernetes.io/projected/887fce08-5e8e-427c-974d-7b2225077411-kube-api-access-bwffq\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.281649 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.281620    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-model-cache\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.281791 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.281659    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-dshm\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.281791 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.281685    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-tmp-dir\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.281791 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.281718    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwffq\" (UniqueName: \"kubernetes.io/projected/887fce08-5e8e-427c-974d-7b2225077411-kube-api-access-bwffq\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.281950 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.281791    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887fce08-5e8e-427c-974d-7b2225077411-tls-certs\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.281950 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.281851    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-home\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.282052 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.282020    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-tmp-dir\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.282111 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.282063    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-model-cache\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.282162 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.282141    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-home\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.284046 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.284027    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-dshm\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.284229 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.284211    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887fce08-5e8e-427c-974d-7b2225077411-tls-certs\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.290522 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.290503    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwffq\" (UniqueName: \"kubernetes.io/projected/887fce08-5e8e-427c-974d-7b2225077411-kube-api-access-bwffq\") pod \"sub-test-model-92482b-kserve-55c5c57dfc-hwwn6\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") " pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.383392 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.383334    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:22.654425 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:22.654397    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"]
Sep 15 22:56:22.656260 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:56:22.656237    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod887fce08_5e8e_427c_974d_7b2225077411.slice/crio-7a6973537c1638886c2d6f540f2e8c6b489721ea810725528312ca8726967b99 WatchSource:0}: Error finding container 7a6973537c1638886c2d6f540f2e8c6b489721ea810725528312ca8726967b99: Status 404 returned error can't find the container with id 7a6973537c1638886c2d6f540f2e8c6b489721ea810725528312ca8726967b99
Sep 15 22:56:23.068226 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:23.068191    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:56:23.566726 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:23.566689    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6" event={"ID":"887fce08-5e8e-427c-974d-7b2225077411","Type":"ContainerStarted","Data":"910c778b2c3791c74d0e408004eee3692a0b346263dc0a0b398352a924372ffd"}
Sep 15 22:56:23.566726 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:23.566731    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6" event={"ID":"887fce08-5e8e-427c-974d-7b2225077411","Type":"ContainerStarted","Data":"7a6973537c1638886c2d6f540f2e8c6b489721ea810725528312ca8726967b99"}
Sep 15 22:56:23.587009 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:23.586958    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6" podStartSLOduration=1.586943324 podStartE2EDuration="1.586943324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:56:23.585581929 +0000 UTC m=+1756.204396838" watchObservedRunningTime="2026-09-15 22:56:23.586943324 +0000 UTC m=+1756.205758239"
Sep 15 22:56:25.770133 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:25.770095    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:56:32.383930 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:32.383897    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:32.384334 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:32.384237    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:32.396602 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:32.396581    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:32.628962 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:32.628931    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:56:41.075509 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:41.075478    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:56:53.455471 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:53.455425    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:56:53.522861 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:53.522818    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:56:55.158657 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:56:55.158626    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:57:02.620070 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:02.619980    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:57:03.803116 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:03.803052    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:57:04.010010 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:04.009972    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"]
Sep 15 22:57:04.056045 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:04.055968    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:57:04.949790 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:04.949760    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"]
Sep 15 22:57:04.971650 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:04.971619    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:57:06.753977 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:06.753940    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:57:07.987942 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:07.987908    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:57:07.990959 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:07.990933    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 22:57:16.688742 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:16.688703    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:57:17.369319 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:17.369281    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:57:19.028628 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.028590    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk" podUID="dd6a3323-22ee-414a-b5f7-b780e25bb16c" containerName="main" containerID="cri-o://34339c5dad77c35afddf8faef7e6a73a5bd56bc26b1bfd8ee999d45427f78291" gracePeriod=45
Sep 15 22:57:19.283322 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.283244    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:57:19.384481 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.384448    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cb4v\" (UniqueName: \"kubernetes.io/projected/dd6a3323-22ee-414a-b5f7-b780e25bb16c-kube-api-access-4cb4v\") pod \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") "
Sep 15 22:57:19.384665 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.384502    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-tmp-dir\") pod \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") "
Sep 15 22:57:19.384665 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.384549    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd6a3323-22ee-414a-b5f7-b780e25bb16c-tls-certs\") pod \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") "
Sep 15 22:57:19.384665 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.384611    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-model-cache\") pod \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") "
Sep 15 22:57:19.384665 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.384645    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-home\") pod \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") "
Sep 15 22:57:19.384878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.384673    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-dshm\") pod \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\" (UID: \"dd6a3323-22ee-414a-b5f7-b780e25bb16c\") "
Sep 15 22:57:19.384878 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.384769    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd6a3323-22ee-414a-b5f7-b780e25bb16c" (UID: "dd6a3323-22ee-414a-b5f7-b780e25bb16c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:57:19.384988 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.384928    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd6a3323-22ee-414a-b5f7-b780e25bb16c" (UID: "dd6a3323-22ee-414a-b5f7-b780e25bb16c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:57:19.384988 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.384947    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-home" (OuterVolumeSpecName: "home") pod "dd6a3323-22ee-414a-b5f7-b780e25bb16c" (UID: "dd6a3323-22ee-414a-b5f7-b780e25bb16c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:57:19.385083 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.384989    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:57:19.386797 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.386762    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd6a3323-22ee-414a-b5f7-b780e25bb16c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd6a3323-22ee-414a-b5f7-b780e25bb16c" (UID: "dd6a3323-22ee-414a-b5f7-b780e25bb16c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:57:19.386797 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.386776    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-dshm" (OuterVolumeSpecName: "dshm") pod "dd6a3323-22ee-414a-b5f7-b780e25bb16c" (UID: "dd6a3323-22ee-414a-b5f7-b780e25bb16c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:57:19.386963 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.386802    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6a3323-22ee-414a-b5f7-b780e25bb16c-kube-api-access-4cb4v" (OuterVolumeSpecName: "kube-api-access-4cb4v") pod "dd6a3323-22ee-414a-b5f7-b780e25bb16c" (UID: "dd6a3323-22ee-414a-b5f7-b780e25bb16c"). InnerVolumeSpecName "kube-api-access-4cb4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:57:19.485917 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.485885    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cb4v\" (UniqueName: \"kubernetes.io/projected/dd6a3323-22ee-414a-b5f7-b780e25bb16c-kube-api-access-4cb4v\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:57:19.485917 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.485912    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd6a3323-22ee-414a-b5f7-b780e25bb16c-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:57:19.486065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.485927    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:57:19.486065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.485939    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:57:19.486065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.485949    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd6a3323-22ee-414a-b5f7-b780e25bb16c-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:57:19.866071 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.866036    2584 generic.go:356] "Generic (PLEG): container finished" podID="dd6a3323-22ee-414a-b5f7-b780e25bb16c" containerID="34339c5dad77c35afddf8faef7e6a73a5bd56bc26b1bfd8ee999d45427f78291" exitCode=0
Sep 15 22:57:19.866300 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.866076    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"
Sep 15 22:57:19.866300 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.866125    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk" event={"ID":"dd6a3323-22ee-414a-b5f7-b780e25bb16c","Type":"ContainerDied","Data":"34339c5dad77c35afddf8faef7e6a73a5bd56bc26b1bfd8ee999d45427f78291"}
Sep 15 22:57:19.866300 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.866173    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk" event={"ID":"dd6a3323-22ee-414a-b5f7-b780e25bb16c","Type":"ContainerDied","Data":"d1074682a1a1809dc1e07d550f649a41bc27d730057705f39bdd36161ef38803"}
Sep 15 22:57:19.866300 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.866196    2584 scope.go:117] "RemoveContainer" containerID="34339c5dad77c35afddf8faef7e6a73a5bd56bc26b1bfd8ee999d45427f78291"
Sep 15 22:57:19.876394 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.876378    2584 scope.go:117] "RemoveContainer" containerID="34339c5dad77c35afddf8faef7e6a73a5bd56bc26b1bfd8ee999d45427f78291"
Sep 15 22:57:19.876673 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:57:19.876654    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34339c5dad77c35afddf8faef7e6a73a5bd56bc26b1bfd8ee999d45427f78291\": container with ID starting with 34339c5dad77c35afddf8faef7e6a73a5bd56bc26b1bfd8ee999d45427f78291 not found: ID does not exist" containerID="34339c5dad77c35afddf8faef7e6a73a5bd56bc26b1bfd8ee999d45427f78291"
Sep 15 22:57:19.876740 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.876685    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34339c5dad77c35afddf8faef7e6a73a5bd56bc26b1bfd8ee999d45427f78291"} err="failed to get container status \"34339c5dad77c35afddf8faef7e6a73a5bd56bc26b1bfd8ee999d45427f78291\": rpc error: code = NotFound desc = could not find container \"34339c5dad77c35afddf8faef7e6a73a5bd56bc26b1bfd8ee999d45427f78291\": container with ID starting with 34339c5dad77c35afddf8faef7e6a73a5bd56bc26b1bfd8ee999d45427f78291 not found: ID does not exist"
Sep 15 22:57:19.887851 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.887828    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"]
Sep 15 22:57:19.889715 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.889686    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0016d4/sub-test-model-0016d4-kserve-c857666cc-p25sk"]
Sep 15 22:57:19.898900 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.898879    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd6a3323-22ee-414a-b5f7-b780e25bb16c" path="/var/lib/kubelet/pods/dd6a3323-22ee-414a-b5f7-b780e25bb16c/volumes"
Sep 15 22:57:19.968308 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:19.968279    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6" podUID="887fce08-5e8e-427c-974d-7b2225077411" containerName="main" containerID="cri-o://910c778b2c3791c74d0e408004eee3692a0b346263dc0a0b398352a924372ffd" gracePeriod=45
Sep 15 22:57:20.228127 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.228099    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:57:20.294218 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.294184    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887fce08-5e8e-427c-974d-7b2225077411-tls-certs\") pod \"887fce08-5e8e-427c-974d-7b2225077411\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") "
Sep 15 22:57:20.294218 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.294223    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-model-cache\") pod \"887fce08-5e8e-427c-974d-7b2225077411\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") "
Sep 15 22:57:20.294495 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.294323    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwffq\" (UniqueName: \"kubernetes.io/projected/887fce08-5e8e-427c-974d-7b2225077411-kube-api-access-bwffq\") pod \"887fce08-5e8e-427c-974d-7b2225077411\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") "
Sep 15 22:57:20.294495 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.294348    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-tmp-dir\") pod \"887fce08-5e8e-427c-974d-7b2225077411\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") "
Sep 15 22:57:20.294495 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.294364    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-home\") pod \"887fce08-5e8e-427c-974d-7b2225077411\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") "
Sep 15 22:57:20.294495 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.294404    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-dshm\") pod \"887fce08-5e8e-427c-974d-7b2225077411\" (UID: \"887fce08-5e8e-427c-974d-7b2225077411\") "
Sep 15 22:57:20.294693 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.294498    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-model-cache" (OuterVolumeSpecName: "model-cache") pod "887fce08-5e8e-427c-974d-7b2225077411" (UID: "887fce08-5e8e-427c-974d-7b2225077411"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:57:20.294693 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.294657    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-home" (OuterVolumeSpecName: "home") pod "887fce08-5e8e-427c-974d-7b2225077411" (UID: "887fce08-5e8e-427c-974d-7b2225077411"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:57:20.294693 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.294671    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "887fce08-5e8e-427c-974d-7b2225077411" (UID: "887fce08-5e8e-427c-974d-7b2225077411"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:57:20.294828 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.294809    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:57:20.294866 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.294836    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:57:20.294866 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.294851    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:57:20.296421 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.296399    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-dshm" (OuterVolumeSpecName: "dshm") pod "887fce08-5e8e-427c-974d-7b2225077411" (UID: "887fce08-5e8e-427c-974d-7b2225077411"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:57:20.296523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.296432    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887fce08-5e8e-427c-974d-7b2225077411-kube-api-access-bwffq" (OuterVolumeSpecName: "kube-api-access-bwffq") pod "887fce08-5e8e-427c-974d-7b2225077411" (UID: "887fce08-5e8e-427c-974d-7b2225077411"). InnerVolumeSpecName "kube-api-access-bwffq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:57:20.296523 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.296506    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887fce08-5e8e-427c-974d-7b2225077411-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "887fce08-5e8e-427c-974d-7b2225077411" (UID: "887fce08-5e8e-427c-974d-7b2225077411"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:57:20.395361 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.395257    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwffq\" (UniqueName: \"kubernetes.io/projected/887fce08-5e8e-427c-974d-7b2225077411-kube-api-access-bwffq\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:57:20.395361 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.395310    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887fce08-5e8e-427c-974d-7b2225077411-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:57:20.395361 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.395321    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887fce08-5e8e-427c-974d-7b2225077411-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:57:20.873257 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.873224    2584 generic.go:356] "Generic (PLEG): container finished" podID="887fce08-5e8e-427c-974d-7b2225077411" containerID="910c778b2c3791c74d0e408004eee3692a0b346263dc0a0b398352a924372ffd" exitCode=0
Sep 15 22:57:20.873432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.873288    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"
Sep 15 22:57:20.873432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.873303    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6" event={"ID":"887fce08-5e8e-427c-974d-7b2225077411","Type":"ContainerDied","Data":"910c778b2c3791c74d0e408004eee3692a0b346263dc0a0b398352a924372ffd"}
Sep 15 22:57:20.873432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.873350    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6" event={"ID":"887fce08-5e8e-427c-974d-7b2225077411","Type":"ContainerDied","Data":"7a6973537c1638886c2d6f540f2e8c6b489721ea810725528312ca8726967b99"}
Sep 15 22:57:20.873432 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.873369    2584 scope.go:117] "RemoveContainer" containerID="910c778b2c3791c74d0e408004eee3692a0b346263dc0a0b398352a924372ffd"
Sep 15 22:57:20.882830 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.882808    2584 scope.go:117] "RemoveContainer" containerID="910c778b2c3791c74d0e408004eee3692a0b346263dc0a0b398352a924372ffd"
Sep 15 22:57:20.883086 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:57:20.883067    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910c778b2c3791c74d0e408004eee3692a0b346263dc0a0b398352a924372ffd\": container with ID starting with 910c778b2c3791c74d0e408004eee3692a0b346263dc0a0b398352a924372ffd not found: ID does not exist" containerID="910c778b2c3791c74d0e408004eee3692a0b346263dc0a0b398352a924372ffd"
Sep 15 22:57:20.883140 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.883094    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910c778b2c3791c74d0e408004eee3692a0b346263dc0a0b398352a924372ffd"} err="failed to get container status \"910c778b2c3791c74d0e408004eee3692a0b346263dc0a0b398352a924372ffd\": rpc error: code = NotFound desc = could not find container \"910c778b2c3791c74d0e408004eee3692a0b346263dc0a0b398352a924372ffd\": container with ID starting with 910c778b2c3791c74d0e408004eee3692a0b346263dc0a0b398352a924372ffd not found: ID does not exist"
Sep 15 22:57:20.894370 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.894348    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"]
Sep 15 22:57:20.898610 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:20.898582    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/sub-test-model-92482b-kserve-55c5c57dfc-hwwn6"]
Sep 15 22:57:21.898191 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:21.898158    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="887fce08-5e8e-427c-974d-7b2225077411" path="/var/lib/kubelet/pods/887fce08-5e8e-427c-974d-7b2225077411/volumes"
Sep 15 22:57:26.161137 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:26.161100    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:57:40.267301 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.267190    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"]
Sep 15 22:57:40.272620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.267602    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd6a3323-22ee-414a-b5f7-b780e25bb16c" containerName="main"
Sep 15 22:57:40.272620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.267616    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd6a3323-22ee-414a-b5f7-b780e25bb16c" containerName="main"
Sep 15 22:57:40.272620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.267631    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887fce08-5e8e-427c-974d-7b2225077411" containerName="main"
Sep 15 22:57:40.272620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.267637    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="887fce08-5e8e-427c-974d-7b2225077411" containerName="main"
Sep 15 22:57:40.272620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.267703    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd6a3323-22ee-414a-b5f7-b780e25bb16c" containerName="main"
Sep 15 22:57:40.272620 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.267712    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="887fce08-5e8e-427c-974d-7b2225077411" containerName="main"
Sep 15 22:57:40.273279 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.273251    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.276091 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.276071    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-92482b\"/\"openshift-service-ca.crt\""
Sep 15 22:57:40.276200 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.276108    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-92482b\"/\"kube-root-ca.crt\""
Sep 15 22:57:40.277436 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.277409    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-92482b\"/\"default-dockercfg-nckqq\""
Sep 15 22:57:40.277697 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.277658    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-92482b\"/\"rate-test-model-92482b-kserve-self-signed-certs\""
Sep 15 22:57:40.280351 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.280309    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"]
Sep 15 22:57:40.294284 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.294225    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:57:40.364101 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.364078    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-home\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.364223 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.364109    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-tmp-dir\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.364223 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.364140    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rd9k\" (UniqueName: \"kubernetes.io/projected/4609ce73-cbea-465e-a051-185ff33e310d-kube-api-access-7rd9k\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.364223 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.364196    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4609ce73-cbea-465e-a051-185ff33e310d-tls-certs\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.364223 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.364221    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-model-cache\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.364425 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.364320    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-dshm\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.464917 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.464891    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-dshm\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.465065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.464926    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-home\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.465065 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.464942    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-tmp-dir\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.465196 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.465075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rd9k\" (UniqueName: \"kubernetes.io/projected/4609ce73-cbea-465e-a051-185ff33e310d-kube-api-access-7rd9k\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.465196 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.465152    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4609ce73-cbea-465e-a051-185ff33e310d-tls-certs\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.465320 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.465210    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-model-cache\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.465384 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.465318    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-tmp-dir\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.465439 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.465377    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-home\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.465617 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.465598    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-model-cache\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.467262 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.467242    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-dshm\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.467677 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.467651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4609ce73-cbea-465e-a051-185ff33e310d-tls-certs\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.473493 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.473466    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rd9k\" (UniqueName: \"kubernetes.io/projected/4609ce73-cbea-465e-a051-185ff33e310d-kube-api-access-7rd9k\") pod \"rate-test-model-92482b-kserve-884d98fc9-xk6cb\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") " pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.584058 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.584025    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:40.855399 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.855369    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"]
Sep 15 22:57:40.857344 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:57:40.857311    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4609ce73_cbea_465e_a051_185ff33e310d.slice/crio-8188d366b7f529643e0c7e32271993740d7bf6407d5bfe328482c7bdac52bae7 WatchSource:0}: Error finding container 8188d366b7f529643e0c7e32271993740d7bf6407d5bfe328482c7bdac52bae7: Status 404 returned error can't find the container with id 8188d366b7f529643e0c7e32271993740d7bf6407d5bfe328482c7bdac52bae7
Sep 15 22:57:40.980299 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.980251    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb" event={"ID":"4609ce73-cbea-465e-a051-185ff33e310d","Type":"ContainerStarted","Data":"be9f464d0e58caeac12111bcceea1f7deb92acc5523a8c797eb10862a3e0c51c"}
Sep 15 22:57:40.980442 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.980308    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb" event={"ID":"4609ce73-cbea-465e-a051-185ff33e310d","Type":"ContainerStarted","Data":"8188d366b7f529643e0c7e32271993740d7bf6407d5bfe328482c7bdac52bae7"}
Sep 15 22:57:40.998806 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:40.998760    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb" podStartSLOduration=0.998745874 podStartE2EDuration="998.745874ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:57:40.997823538 +0000 UTC m=+1833.616638446" watchObservedRunningTime="2026-09-15 22:57:40.998745874 +0000 UTC m=+1833.617560782"
Sep 15 22:57:43.869808 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:43.869772    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:57:48.055611 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:48.055581    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:57:50.584780 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:50.584744    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:50.585154 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:50.584799    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:50.597889 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:50.597864    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:51.040230 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:51.040154    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:57:58.057165 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:57:58.057115    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:58:08.869960 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:08.869928    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:58:13.865426 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:13.865388    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:14.839625 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:14.839591    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"]
Sep 15 22:58:14.857786 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:14.857755    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:15.293027 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.292939    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:15.569019 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.568987    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"]
Sep 15 22:58:15.572564 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.572543    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.575593 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.575569    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-637a6f-openshift-default-dockercfg-rlbgd\""
Sep 15 22:58:15.587839 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.587817    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"]
Sep 15 22:58:15.598952 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.598924    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:15.660072 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.660038    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-istio-envoy\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.660234 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.660086    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51454f48-4753-4ebf-9979-26f01e42e967-istio-podinfo\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.660234 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.660179    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-credential-socket\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.660234 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.660209    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-istio-data\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.660444 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.660238    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-workload-certs\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.660444 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.660254    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51454f48-4753-4ebf-9979-26f01e42e967-istio-token\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.660444 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.660346    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljglt\" (UniqueName: \"kubernetes.io/projected/51454f48-4753-4ebf-9979-26f01e42e967-kube-api-access-ljglt\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.660621 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.660436    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51454f48-4753-4ebf-9979-26f01e42e967-istiod-ca-cert\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.660621 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.660481    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-workload-socket\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.761222 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.761183    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51454f48-4753-4ebf-9979-26f01e42e967-istio-podinfo\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.761409 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.761255    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-credential-socket\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.761409 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.761306    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-istio-data\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.761409 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.761346    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-workload-certs\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.761409 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.761369    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51454f48-4753-4ebf-9979-26f01e42e967-istio-token\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.761409 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.761396    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljglt\" (UniqueName: \"kubernetes.io/projected/51454f48-4753-4ebf-9979-26f01e42e967-kube-api-access-ljglt\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.761675 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.761440    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51454f48-4753-4ebf-9979-26f01e42e967-istiod-ca-cert\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.761675 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.761474    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-workload-socket\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.761675 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.761524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-istio-envoy\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.762067 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.762040    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-credential-socket\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.762261 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.762111    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-workload-socket\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.762261 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.762146    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-workload-certs\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.762450 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.762182    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-istio-data\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.762505 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.762443    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51454f48-4753-4ebf-9979-26f01e42e967-istiod-ca-cert\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.763942 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.763924    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-istio-envoy\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.764165 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.764148    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51454f48-4753-4ebf-9979-26f01e42e967-istio-podinfo\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.770209 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.770181    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51454f48-4753-4ebf-9979-26f01e42e967-istio-token\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.770445 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.770426    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljglt\" (UniqueName: \"kubernetes.io/projected/51454f48-4753-4ebf-9979-26f01e42e967-kube-api-access-ljglt\") pod \"e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") " pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:15.883402 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:15.883325    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:16.163975 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:16.163948    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"]
Sep 15 22:58:16.165910 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:58:16.165883    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51454f48_4753_4ebf_9979_26f01e42e967.slice/crio-97681c56dd2dac6a230a460b16b518de36559ca035759be458d7d8d56190af2f WatchSource:0}: Error finding container 97681c56dd2dac6a230a460b16b518de36559ca035759be458d7d8d56190af2f: Status 404 returned error can't find the container with id 97681c56dd2dac6a230a460b16b518de36559ca035759be458d7d8d56190af2f
Sep 15 22:58:16.168205 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:16.168164    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:58:16.168328 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:16.168246    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:58:16.168328 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:16.168301    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:58:16.365824 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:16.365787    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:17.161866 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:17.161827    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" event={"ID":"51454f48-4753-4ebf-9979-26f01e42e967","Type":"ContainerStarted","Data":"5bc3bc66b141298672362cac159445e7895c80bad5b44b70a3aaef289d161e8c"}
Sep 15 22:58:17.162079 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:17.161885    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" event={"ID":"51454f48-4753-4ebf-9979-26f01e42e967","Type":"ContainerStarted","Data":"97681c56dd2dac6a230a460b16b518de36559ca035759be458d7d8d56190af2f"}
Sep 15 22:58:17.185695 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:17.185650    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" podStartSLOduration=2.1856374450000002 podStartE2EDuration="2.185637445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:58:17.183348067 +0000 UTC m=+1869.802162974" watchObservedRunningTime="2026-09-15 22:58:17.185637445 +0000 UTC m=+1869.804452408"
Sep 15 22:58:17.883606 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:17.883566    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:17.889060 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:17.889034    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:18.166680 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:18.166591    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:18.167570 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:18.167548    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:18.934206 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:18.934172    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:19.104311 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:19.104258    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:19.199668 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:19.199574    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:19.924216 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:19.924184    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:20.781794 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:20.781756    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:20.891949 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:20.891915    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:21.047723 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.047691    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:21.468327 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.468238    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:21.573673 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.573642    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:21.676512 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.676479    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"]
Sep 15 22:58:21.680147 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.680129    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"
Sep 15 22:58:21.683114 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.683092    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-637a6f-dockercfg-zvtq4\""
Sep 15 22:58:21.683232 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.683095    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-637a6f\""
Sep 15 22:58:21.691619 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.691596    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"]
Sep 15 22:58:21.773660 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.773587    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"]
Sep 15 22:58:21.777118 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.777099    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"
Sep 15 22:58:21.786601 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.786578    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"]
Sep 15 22:58:21.822771 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.822740    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx4qw\" (UniqueName: \"kubernetes.io/projected/9d2269c2-3c50-4443-9736-ba6187567366-kube-api-access-kx4qw\") pod \"payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p\" (UID: \"9d2269c2-3c50-4443-9736-ba6187567366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"
Sep 15 22:58:21.822907 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.822781    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2269c2-3c50-4443-9736-ba6187567366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p\" (UID: \"9d2269c2-3c50-4443-9736-ba6187567366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"
Sep 15 22:58:21.923759 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.923728    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx4qw\" (UniqueName: \"kubernetes.io/projected/9d2269c2-3c50-4443-9736-ba6187567366-kube-api-access-kx4qw\") pod \"payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p\" (UID: \"9d2269c2-3c50-4443-9736-ba6187567366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"
Sep 15 22:58:21.923759 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.923762    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tgrx\" (UniqueName: \"kubernetes.io/projected/fe778af7-75f2-44ec-aa47-c089df42f5d5-kube-api-access-6tgrx\") pod \"payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx\" (UID: \"fe778af7-75f2-44ec-aa47-c089df42f5d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"
Sep 15 22:58:21.924004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.923787    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2269c2-3c50-4443-9736-ba6187567366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p\" (UID: \"9d2269c2-3c50-4443-9736-ba6187567366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"
Sep 15 22:58:21.924004 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.923921    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe778af7-75f2-44ec-aa47-c089df42f5d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx\" (UID: \"fe778af7-75f2-44ec-aa47-c089df42f5d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"
Sep 15 22:58:21.924501 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.924483    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2269c2-3c50-4443-9736-ba6187567366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p\" (UID: \"9d2269c2-3c50-4443-9736-ba6187567366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"
Sep 15 22:58:21.935647 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.935621    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx4qw\" (UniqueName: \"kubernetes.io/projected/9d2269c2-3c50-4443-9736-ba6187567366-kube-api-access-kx4qw\") pod \"payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p\" (UID: \"9d2269c2-3c50-4443-9736-ba6187567366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"
Sep 15 22:58:21.991316 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:21.991290    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"
Sep 15 22:58:22.024745 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:22.024661    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tgrx\" (UniqueName: \"kubernetes.io/projected/fe778af7-75f2-44ec-aa47-c089df42f5d5-kube-api-access-6tgrx\") pod \"payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx\" (UID: \"fe778af7-75f2-44ec-aa47-c089df42f5d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"
Sep 15 22:58:22.024745 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:22.024727    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe778af7-75f2-44ec-aa47-c089df42f5d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx\" (UID: \"fe778af7-75f2-44ec-aa47-c089df42f5d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"
Sep 15 22:58:22.025488 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:22.025467    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe778af7-75f2-44ec-aa47-c089df42f5d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx\" (UID: \"fe778af7-75f2-44ec-aa47-c089df42f5d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"
Sep 15 22:58:22.035701 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:22.035678    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tgrx\" (UniqueName: \"kubernetes.io/projected/fe778af7-75f2-44ec-aa47-c089df42f5d5-kube-api-access-6tgrx\") pod \"payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx\" (UID: \"fe778af7-75f2-44ec-aa47-c089df42f5d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"
Sep 15 22:58:22.089761 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:22.089725    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"
Sep 15 22:58:22.265823 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:22.265798    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"]
Sep 15 22:58:22.267792 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:58:22.267766    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2269c2_3c50_4443_9736_ba6187567366.slice/crio-ca08027d87d405e8360dce3e155ab250258eb301486bfd1d124c5261100bf4c0 WatchSource:0}: Error finding container ca08027d87d405e8360dce3e155ab250258eb301486bfd1d124c5261100bf4c0: Status 404 returned error can't find the container with id ca08027d87d405e8360dce3e155ab250258eb301486bfd1d124c5261100bf4c0
Sep 15 22:58:22.367711 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:22.367682    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"]
Sep 15 22:58:22.370674 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:58:22.370651    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe778af7_75f2_44ec_aa47_c089df42f5d5.slice/crio-fefb65bf6d81acddd7ad0df03884b6696c34fd0b2028855943c64e941994a50f WatchSource:0}: Error finding container fefb65bf6d81acddd7ad0df03884b6696c34fd0b2028855943c64e941994a50f: Status 404 returned error can't find the container with id fefb65bf6d81acddd7ad0df03884b6696c34fd0b2028855943c64e941994a50f
Sep 15 22:58:23.198076 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:23.198025    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx" event={"ID":"fe778af7-75f2-44ec-aa47-c089df42f5d5","Type":"ContainerStarted","Data":"24c9546b9fd52dd2ba87b9b48151553c2308ae4ceb6299513a1445ebe4ec41fd"}
Sep 15 22:58:23.198572 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:23.198102    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"
Sep 15 22:58:23.198572 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:23.198126    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx" event={"ID":"fe778af7-75f2-44ec-aa47-c089df42f5d5","Type":"ContainerStarted","Data":"fefb65bf6d81acddd7ad0df03884b6696c34fd0b2028855943c64e941994a50f"}
Sep 15 22:58:23.199690 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:23.199663    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p" event={"ID":"9d2269c2-3c50-4443-9736-ba6187567366","Type":"ContainerStarted","Data":"d58d9efbc482911f41ef8ecf310c78e4c0586b7c904e57f8ce308b986cc5f830"}
Sep 15 22:58:23.199826 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:23.199710    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p" event={"ID":"9d2269c2-3c50-4443-9736-ba6187567366","Type":"ContainerStarted","Data":"ca08027d87d405e8360dce3e155ab250258eb301486bfd1d124c5261100bf4c0"}
Sep 15 22:58:23.199826 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:23.199739    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"
Sep 15 22:58:23.221743 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:23.221669    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx" podStartSLOduration=2.221650691 podStartE2EDuration="2.221650691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:58:23.219120531 +0000 UTC m=+1875.837935443" watchObservedRunningTime="2026-09-15 22:58:23.221650691 +0000 UTC m=+1875.840465604"
Sep 15 22:58:23.242508 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:23.242464    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p" podStartSLOduration=2.242449633 podStartE2EDuration="2.242449633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:58:23.239953983 +0000 UTC m=+1875.858768890" watchObservedRunningTime="2026-09-15 22:58:23.242449633 +0000 UTC m=+1875.861264554"
Sep 15 22:58:24.361991 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:24.361955    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:58:28.617351 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:28.617305    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:29.859536 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:29.859492    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb" podUID="4609ce73-cbea-465e-a051-185ff33e310d" containerName="main" containerID="cri-o://be9f464d0e58caeac12111bcceea1f7deb92acc5523a8c797eb10862a3e0c51c" gracePeriod=45
Sep 15 22:58:30.139021 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.138997    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:58:30.236776 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.236695    2584 generic.go:356] "Generic (PLEG): container finished" podID="4609ce73-cbea-465e-a051-185ff33e310d" containerID="be9f464d0e58caeac12111bcceea1f7deb92acc5523a8c797eb10862a3e0c51c" exitCode=0
Sep 15 22:58:30.236776 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.236740    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"
Sep 15 22:58:30.237017 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.236787    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb" event={"ID":"4609ce73-cbea-465e-a051-185ff33e310d","Type":"ContainerDied","Data":"be9f464d0e58caeac12111bcceea1f7deb92acc5523a8c797eb10862a3e0c51c"}
Sep 15 22:58:30.237017 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.236836    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb" event={"ID":"4609ce73-cbea-465e-a051-185ff33e310d","Type":"ContainerDied","Data":"8188d366b7f529643e0c7e32271993740d7bf6407d5bfe328482c7bdac52bae7"}
Sep 15 22:58:30.237017 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.236862    2584 scope.go:117] "RemoveContainer" containerID="be9f464d0e58caeac12111bcceea1f7deb92acc5523a8c797eb10862a3e0c51c"
Sep 15 22:58:30.245496 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.245480    2584 scope.go:117] "RemoveContainer" containerID="be9f464d0e58caeac12111bcceea1f7deb92acc5523a8c797eb10862a3e0c51c"
Sep 15 22:58:30.245776 ip-10-0-138-116 kubenswrapper[2584]: E0915 22:58:30.245747    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9f464d0e58caeac12111bcceea1f7deb92acc5523a8c797eb10862a3e0c51c\": container with ID starting with be9f464d0e58caeac12111bcceea1f7deb92acc5523a8c797eb10862a3e0c51c not found: ID does not exist" containerID="be9f464d0e58caeac12111bcceea1f7deb92acc5523a8c797eb10862a3e0c51c"
Sep 15 22:58:30.245830 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.245785    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9f464d0e58caeac12111bcceea1f7deb92acc5523a8c797eb10862a3e0c51c"} err="failed to get container status \"be9f464d0e58caeac12111bcceea1f7deb92acc5523a8c797eb10862a3e0c51c\": rpc error: code = NotFound desc = could not find container \"be9f464d0e58caeac12111bcceea1f7deb92acc5523a8c797eb10862a3e0c51c\": container with ID starting with be9f464d0e58caeac12111bcceea1f7deb92acc5523a8c797eb10862a3e0c51c not found: ID does not exist"
Sep 15 22:58:30.311118 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.311084    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rd9k\" (UniqueName: \"kubernetes.io/projected/4609ce73-cbea-465e-a051-185ff33e310d-kube-api-access-7rd9k\") pod \"4609ce73-cbea-465e-a051-185ff33e310d\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") "
Sep 15 22:58:30.311302 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.311146    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-tmp-dir\") pod \"4609ce73-cbea-465e-a051-185ff33e310d\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") "
Sep 15 22:58:30.311302 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.311169    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-home\") pod \"4609ce73-cbea-465e-a051-185ff33e310d\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") "
Sep 15 22:58:30.311302 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.311195    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-model-cache\") pod \"4609ce73-cbea-465e-a051-185ff33e310d\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") "
Sep 15 22:58:30.311449 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.311330    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4609ce73-cbea-465e-a051-185ff33e310d-tls-certs\") pod \"4609ce73-cbea-465e-a051-185ff33e310d\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") "
Sep 15 22:58:30.311449 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.311367    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-dshm\") pod \"4609ce73-cbea-465e-a051-185ff33e310d\" (UID: \"4609ce73-cbea-465e-a051-185ff33e310d\") "
Sep 15 22:58:30.311449 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.311419    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-home" (OuterVolumeSpecName: "home") pod "4609ce73-cbea-465e-a051-185ff33e310d" (UID: "4609ce73-cbea-465e-a051-185ff33e310d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:58:30.311579 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.311428    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4609ce73-cbea-465e-a051-185ff33e310d" (UID: "4609ce73-cbea-465e-a051-185ff33e310d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:58:30.311579 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.311509    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-model-cache" (OuterVolumeSpecName: "model-cache") pod "4609ce73-cbea-465e-a051-185ff33e310d" (UID: "4609ce73-cbea-465e-a051-185ff33e310d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:58:30.311720 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.311701    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:58:30.311766 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.311726    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:58:30.311766 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.311741    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:58:30.313278 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.313236    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4609ce73-cbea-465e-a051-185ff33e310d-kube-api-access-7rd9k" (OuterVolumeSpecName: "kube-api-access-7rd9k") pod "4609ce73-cbea-465e-a051-185ff33e310d" (UID: "4609ce73-cbea-465e-a051-185ff33e310d"). InnerVolumeSpecName "kube-api-access-7rd9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:58:30.313409 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.313303    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4609ce73-cbea-465e-a051-185ff33e310d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4609ce73-cbea-465e-a051-185ff33e310d" (UID: "4609ce73-cbea-465e-a051-185ff33e310d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:58:30.313508 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.313483    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-dshm" (OuterVolumeSpecName: "dshm") pod "4609ce73-cbea-465e-a051-185ff33e310d" (UID: "4609ce73-cbea-465e-a051-185ff33e310d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:58:30.412421 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.412334    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4609ce73-cbea-465e-a051-185ff33e310d-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:58:30.412421 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.412364    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4609ce73-cbea-465e-a051-185ff33e310d-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:58:30.412421 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.412375    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rd9k\" (UniqueName: \"kubernetes.io/projected/4609ce73-cbea-465e-a051-185ff33e310d-kube-api-access-7rd9k\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 22:58:30.560147 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.560114    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"]
Sep 15 22:58:30.564823 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:30.564793    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-92482b/rate-test-model-92482b-kserve-884d98fc9-xk6cb"]
Sep 15 22:58:31.899697 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:31.899658    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4609ce73-cbea-465e-a051-185ff33e310d" path="/var/lib/kubelet/pods/4609ce73-cbea-465e-a051-185ff33e310d/volumes"
Sep 15 22:58:34.002137 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:34.001572    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": read tcp 10.132.0.2:33646->10.132.0.98:15021: read: connection reset by peer" start-of-body=
Sep 15 22:58:34.002137 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:34.001637    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" podUID="51454f48-4753-4ebf-9979-26f01e42e967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": read tcp 10.132.0.2:33646->10.132.0.98:15021: read: connection reset by peer"
Sep 15 22:58:34.207396 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:34.207325    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"
Sep 15 22:58:34.207396 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:34.207390    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"
Sep 15 22:58:34.263484 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:34.263404    2584 generic.go:356] "Generic (PLEG): container finished" podID="51454f48-4753-4ebf-9979-26f01e42e967" containerID="5bc3bc66b141298672362cac159445e7895c80bad5b44b70a3aaef289d161e8c" exitCode=0
Sep 15 22:58:34.263484 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:34.263461    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" event={"ID":"51454f48-4753-4ebf-9979-26f01e42e967","Type":"ContainerDied","Data":"5bc3bc66b141298672362cac159445e7895c80bad5b44b70a3aaef289d161e8c"}
Sep 15 22:58:34.263866 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:34.263852    2584 scope.go:117] "RemoveContainer" containerID="5bc3bc66b141298672362cac159445e7895c80bad5b44b70a3aaef289d161e8c"
Sep 15 22:58:34.264988 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:34.264959    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:58:34.265087 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:34.265036    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:58:34.265087 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:34.265071    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:58:35.283289 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:35.274639    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" event={"ID":"51454f48-4753-4ebf-9979-26f01e42e967","Type":"ContainerStarted","Data":"31636ee71f25902d9ee50cef7188ae8fbcbd386e7b707c0af2cf178c12c7cc47"}
Sep 15 22:58:35.884603 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:35.884556    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:35.884817 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:35.884694    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 15 22:58:35.884817 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:35.884741    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" podUID="51454f48-4753-4ebf-9979-26f01e42e967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 15 22:58:36.884129 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:36.884088    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 15 22:58:36.884744 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:36.884710    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" podUID="51454f48-4753-4ebf-9979-26f01e42e967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 15 22:58:37.884885 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:37.884842    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 15 22:58:37.885495 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:37.885458    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" podUID="51454f48-4753-4ebf-9979-26f01e42e967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 15 22:58:38.884337 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:38.884294    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 15 22:58:38.884635 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:38.884606    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" podUID="51454f48-4753-4ebf-9979-26f01e42e967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 15 22:58:39.884137 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:39.884096    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 15 22:58:39.884722 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:39.884691    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" podUID="51454f48-4753-4ebf-9979-26f01e42e967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 15 22:58:40.888001 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:40.887967    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:40.888406 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:40.888294    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:40.888889 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:40.888872    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 22:58:43.151256 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:43.151201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:43.856631 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:43.856597    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:44.092896 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:44.092860    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:44.543768 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:44.543735    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:44.731168 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:44.731099    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:45.007820 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.007780    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"]
Sep 15 22:58:45.008648 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.008618    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4609ce73-cbea-465e-a051-185ff33e310d" containerName="main"
Sep 15 22:58:45.008648 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.008647    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4609ce73-cbea-465e-a051-185ff33e310d" containerName="main"
Sep 15 22:58:45.008838 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.008795    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4609ce73-cbea-465e-a051-185ff33e310d" containerName="main"
Sep 15 22:58:45.012124 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.012102    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.015356 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.015334    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-668e8d-openshift-default-dockercfg-mjh79\""
Sep 15 22:58:45.026927 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.026903    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"]
Sep 15 22:58:45.045422 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.045398    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:45.051932 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.051905    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-credential-socket\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.052062 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.051943    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ac9170-704f-4f09-be27-f875af3abe20-istio-podinfo\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.052131 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.052067    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ac9170-704f-4f09-be27-f875af3abe20-istio-token\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.052131 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.052113    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-workload-socket\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.052231 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.052178    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgpwd\" (UniqueName: \"kubernetes.io/projected/32ac9170-704f-4f09-be27-f875af3abe20-kube-api-access-pgpwd\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.052296 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.052248    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ac9170-704f-4f09-be27-f875af3abe20-istiod-ca-cert\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.052348 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.052306    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-istio-envoy\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.052348 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.052337    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-workload-certs\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.052430 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.052380    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-istio-data\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.153603 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.153569    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ac9170-704f-4f09-be27-f875af3abe20-istio-token\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.153773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.153622    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-workload-socket\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.153773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.153668    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpwd\" (UniqueName: \"kubernetes.io/projected/32ac9170-704f-4f09-be27-f875af3abe20-kube-api-access-pgpwd\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.153773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.153709    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ac9170-704f-4f09-be27-f875af3abe20-istiod-ca-cert\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.153773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.153743    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-istio-envoy\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.153773 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.153768    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-workload-certs\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.153999 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.153811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-istio-data\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.153999 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.153850    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-credential-socket\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.153999 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.153875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ac9170-704f-4f09-be27-f875af3abe20-istio-podinfo\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.154252 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.154225    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-workload-certs\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.154414 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.154308    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-credential-socket\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.154414 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.154367    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-workload-socket\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.154414 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.154365    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-istio-data\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.154592 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.154460    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ac9170-704f-4f09-be27-f875af3abe20-istiod-ca-cert\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.156206 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.156184    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-istio-envoy\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.156428 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.156406    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ac9170-704f-4f09-be27-f875af3abe20-istio-podinfo\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.162643 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.162623    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ac9170-704f-4f09-be27-f875af3abe20-istio-token\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.163061 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.163038    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpwd\" (UniqueName: \"kubernetes.io/projected/32ac9170-704f-4f09-be27-f875af3abe20-kube-api-access-pgpwd\") pod \"e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") " pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.324503 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.324471    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:45.387779 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.387243    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:45.487249 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.487217    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:45.603166 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.603138    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"]
Sep 15 22:58:45.604450 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:58:45.604422    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ac9170_704f_4f09_be27_f875af3abe20.slice/crio-d7df0019a19bf33b1a5f2bf3ffe326fc2c1d0dbc26388e06d67512fbbe418ed9 WatchSource:0}: Error finding container d7df0019a19bf33b1a5f2bf3ffe326fc2c1d0dbc26388e06d67512fbbe418ed9: Status 404 returned error can't find the container with id d7df0019a19bf33b1a5f2bf3ffe326fc2c1d0dbc26388e06d67512fbbe418ed9
Sep 15 22:58:45.606613 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.606578    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:58:45.606717 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.606660    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:58:45.606717 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:45.606709    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:58:46.345030 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:46.344993    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" event={"ID":"32ac9170-704f-4f09-be27-f875af3abe20","Type":"ContainerStarted","Data":"7c45018205097481f6d6904aac0b142cb5f5033329e1936b3d1e42f3b36609a4"}
Sep 15 22:58:46.345216 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:46.345075    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" event={"ID":"32ac9170-704f-4f09-be27-f875af3abe20","Type":"ContainerStarted","Data":"d7df0019a19bf33b1a5f2bf3ffe326fc2c1d0dbc26388e06d67512fbbe418ed9"}
Sep 15 22:58:46.369068 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:46.369014    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" podStartSLOduration=2.368998038 podStartE2EDuration="2.368998038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:58:46.367582837 +0000 UTC m=+1898.986397745" watchObservedRunningTime="2026-09-15 22:58:46.368998038 +0000 UTC m=+1898.987812947"
Sep 15 22:58:47.324791 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:47.324759    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:47.329858 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:47.329831    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:47.351626 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:47.351599    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:47.352616 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:47.352594    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:58:47.577457 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:47.577367    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:48.265346 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:48.265309    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:48.356187 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:48.356153    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:48.532494 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:48.532407    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:49.392887 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:49.392857    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:50.265588 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:50.265549    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:51.081385 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:51.081349    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:51.181603 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:51.181562    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:51.236402 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:51.236370    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:51.563890 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:51.563853    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:51.647885 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:51.647844    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:58:55.964081 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:58:55.964048    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:59:03.393476 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:03.393424    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 22:59:03.394013 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:03.393507    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" podUID="32ac9170-704f-4f09-be27-f875af3abe20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 22:59:06.456753 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:06.456715    2584 generic.go:356] "Generic (PLEG): container finished" podID="32ac9170-704f-4f09-be27-f875af3abe20" containerID="7c45018205097481f6d6904aac0b142cb5f5033329e1936b3d1e42f3b36609a4" exitCode=0
Sep 15 22:59:06.457166 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:06.456788    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" event={"ID":"32ac9170-704f-4f09-be27-f875af3abe20","Type":"ContainerDied","Data":"7c45018205097481f6d6904aac0b142cb5f5033329e1936b3d1e42f3b36609a4"}
Sep 15 22:59:06.457210 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:06.457184    2584 scope.go:117] "RemoveContainer" containerID="7c45018205097481f6d6904aac0b142cb5f5033329e1936b3d1e42f3b36609a4"
Sep 15 22:59:06.458370 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:06.458340    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:59:06.458447 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:06.458416    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:59:06.458526 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:06.458448    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 22:59:07.464290 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:07.464239    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" event={"ID":"32ac9170-704f-4f09-be27-f875af3abe20","Type":"ContainerStarted","Data":"25c60cbfc6dabc53cf7a51ec039ba7c86be818f238af5e9b00af868a5ea4bf20"}
Sep 15 22:59:08.324848 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:08.324810    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:59:08.325032 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:08.324989    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Sep 15 22:59:08.325084 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:08.325046    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" podUID="32ac9170-704f-4f09-be27-f875af3abe20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Sep 15 22:59:09.325699 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:09.325668    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Sep 15 22:59:09.326060 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:09.325722    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" podUID="32ac9170-704f-4f09-be27-f875af3abe20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Sep 15 22:59:10.325110 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:10.325076    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Sep 15 22:59:10.325311 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:10.325131    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" podUID="32ac9170-704f-4f09-be27-f875af3abe20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Sep 15 22:59:11.325138 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:11.325100    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Sep 15 22:59:11.325557 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:11.325160    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" podUID="32ac9170-704f-4f09-be27-f875af3abe20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Sep 15 22:59:12.325531 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:12.325493    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Sep 15 22:59:12.325908 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:12.325551    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" podUID="32ac9170-704f-4f09-be27-f875af3abe20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Sep 15 22:59:13.246489 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:13.246449    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:13.329193 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:13.329162    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:59:13.329593 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:13.329466    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:59:13.330172 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:13.330152    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 22:59:13.352194 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:13.352162    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:14.066309 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:14.066251    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:14.773415 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:14.773389    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:15.478746 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:15.478709    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:16.281508 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:16.281476    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:18.254938 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:18.254907    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x6hfn_af0f1d9d-0d60-4861-85c6-e991c8e58bd7/payload-processing/0.log"
Sep 15 22:59:23.788683 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:23.788628    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:26.807716 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:26.807679    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:27.961160 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:27.961128    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:59:42.764555 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:42.764518    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 22:59:48.048628 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:48.048592    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:49.188817 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:49.188784    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx_fe778af7-75f2-44ec-aa47-c089df42f5d5/payload-processing/0.log"
Sep 15 22:59:49.299288 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:49.299236    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x6hfn_af0f1d9d-0d60-4861-85c6-e991c8e58bd7/payload-processing/0.log"
Sep 15 22:59:49.675672 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:49.675635    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:49.988948 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:49.988855    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:50.140856 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:50.140816    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:51.715934 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:51.715899    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:51.770307 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:51.770259    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:52.377111 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:52.377073    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:53.156433 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:53.156395    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:53.256186 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:53.256154    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:53.988913 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:53.988877    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:54.777681 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:54.777641    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:55.466155 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:55.466114    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:56.229235 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:56.229189    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:56.327490 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:56.327452    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:56.387693 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:56.387660    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:56.806874 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:56.806839    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:56.903854 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:56.903813    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 22:59:56.939899 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:56.939844    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"]
Sep 15 22:59:56.943621 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:56.943602    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"
Sep 15 22:59:56.946217 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:56.946192    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-10ec22\""
Sep 15 22:59:56.946363 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:56.946343    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-10ec22-dockercfg-jgm55\""
Sep 15 22:59:56.953169 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:56.953147    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"]
Sep 15 22:59:56.973789 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:56.973761    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"]
Sep 15 22:59:56.977212 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:56.977196    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"
Sep 15 22:59:56.987476 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:56.987451    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"]
Sep 15 22:59:57.044776 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.044740    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vmq2\" (UniqueName: \"kubernetes.io/projected/1be39ce3-ffd0-4d25-bf14-2c29292d5459-kube-api-access-4vmq2\") pod \"payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l\" (UID: \"1be39ce3-ffd0-4d25-bf14-2c29292d5459\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"
Sep 15 22:59:57.044776 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.044779    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xmpg\" (UniqueName: \"kubernetes.io/projected/b878802c-c763-41a0-be2e-e5365010b5cb-kube-api-access-5xmpg\") pod \"payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b\" (UID: \"b878802c-c763-41a0-be2e-e5365010b5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"
Sep 15 22:59:57.045052 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.044918    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be39ce3-ffd0-4d25-bf14-2c29292d5459-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l\" (UID: \"1be39ce3-ffd0-4d25-bf14-2c29292d5459\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"
Sep 15 22:59:57.045052 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.044947    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b878802c-c763-41a0-be2e-e5365010b5cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b\" (UID: \"b878802c-c763-41a0-be2e-e5365010b5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"
Sep 15 22:59:57.145894 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.145798    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be39ce3-ffd0-4d25-bf14-2c29292d5459-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l\" (UID: \"1be39ce3-ffd0-4d25-bf14-2c29292d5459\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"
Sep 15 22:59:57.145894 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.145856    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b878802c-c763-41a0-be2e-e5365010b5cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b\" (UID: \"b878802c-c763-41a0-be2e-e5365010b5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"
Sep 15 22:59:57.146129 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.145917    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vmq2\" (UniqueName: \"kubernetes.io/projected/1be39ce3-ffd0-4d25-bf14-2c29292d5459-kube-api-access-4vmq2\") pod \"payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l\" (UID: \"1be39ce3-ffd0-4d25-bf14-2c29292d5459\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"
Sep 15 22:59:57.146129 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.145943    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xmpg\" (UniqueName: \"kubernetes.io/projected/b878802c-c763-41a0-be2e-e5365010b5cb-kube-api-access-5xmpg\") pod \"payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b\" (UID: \"b878802c-c763-41a0-be2e-e5365010b5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"
Sep 15 22:59:57.146755 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.146734    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be39ce3-ffd0-4d25-bf14-2c29292d5459-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l\" (UID: \"1be39ce3-ffd0-4d25-bf14-2c29292d5459\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"
Sep 15 22:59:57.146833 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.146792    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b878802c-c763-41a0-be2e-e5365010b5cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b\" (UID: \"b878802c-c763-41a0-be2e-e5365010b5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"
Sep 15 22:59:57.155046 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.154935    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vmq2\" (UniqueName: \"kubernetes.io/projected/1be39ce3-ffd0-4d25-bf14-2c29292d5459-kube-api-access-4vmq2\") pod \"payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l\" (UID: \"1be39ce3-ffd0-4d25-bf14-2c29292d5459\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"
Sep 15 22:59:57.161213 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.161188    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xmpg\" (UniqueName: \"kubernetes.io/projected/b878802c-c763-41a0-be2e-e5365010b5cb-kube-api-access-5xmpg\") pod \"payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b\" (UID: \"b878802c-c763-41a0-be2e-e5365010b5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"
Sep 15 22:59:57.255320 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.255256    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"
Sep 15 22:59:57.288373 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.288337    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"
Sep 15 22:59:57.535874 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.535847    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"]
Sep 15 22:59:57.538111 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:59:57.538082    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be39ce3_ffd0_4d25_bf14_2c29292d5459.slice/crio-c3c31c9ba433a24e096a0a5e922a76b34ba4d36b661d185da5fda8232aba042d WatchSource:0}: Error finding container c3c31c9ba433a24e096a0a5e922a76b34ba4d36b661d185da5fda8232aba042d: Status 404 returned error can't find the container with id c3c31c9ba433a24e096a0a5e922a76b34ba4d36b661d185da5fda8232aba042d
Sep 15 22:59:57.572010 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.571545    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"]
Sep 15 22:59:57.573686 ip-10-0-138-116 kubenswrapper[2584]: W0915 22:59:57.573660    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb878802c_c763_41a0_be2e_e5365010b5cb.slice/crio-71c65ce72e63ee768cda81db6e8a99d8f90a6d05c7cf544f4d3c590f7c40e828 WatchSource:0}: Error finding container 71c65ce72e63ee768cda81db6e8a99d8f90a6d05c7cf544f4d3c590f7c40e828: Status 404 returned error can't find the container with id 71c65ce72e63ee768cda81db6e8a99d8f90a6d05c7cf544f4d3c590f7c40e828
Sep 15 22:59:57.744338 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.744305    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l" event={"ID":"1be39ce3-ffd0-4d25-bf14-2c29292d5459","Type":"ContainerStarted","Data":"9922ee812f5650bcd7458ae10a69373ded94f9336df3c41b279c19201484df79"}
Sep 15 22:59:57.744503 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.744364    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l" event={"ID":"1be39ce3-ffd0-4d25-bf14-2c29292d5459","Type":"ContainerStarted","Data":"c3c31c9ba433a24e096a0a5e922a76b34ba4d36b661d185da5fda8232aba042d"}
Sep 15 22:59:57.744602 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.744526    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"
Sep 15 22:59:57.746136 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.746111    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b" event={"ID":"b878802c-c763-41a0-be2e-e5365010b5cb","Type":"ContainerStarted","Data":"16daf934e8a355d651d5a5b61b11d1a60f7f0a89a349ca816c8759da2a18e09b"}
Sep 15 22:59:57.746248 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.746156    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b" event={"ID":"b878802c-c763-41a0-be2e-e5365010b5cb","Type":"ContainerStarted","Data":"71c65ce72e63ee768cda81db6e8a99d8f90a6d05c7cf544f4d3c590f7c40e828"}
Sep 15 22:59:57.746248 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.746178    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"
Sep 15 22:59:57.761145 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.761104    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l" podStartSLOduration=1.761090338 podStartE2EDuration="1.761090338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:59:57.759636843 +0000 UTC m=+1970.378451751" watchObservedRunningTime="2026-09-15 22:59:57.761090338 +0000 UTC m=+1970.379905243"
Sep 15 22:59:57.779617 ip-10-0-138-116 kubenswrapper[2584]: I0915 22:59:57.779566    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b" podStartSLOduration=1.779547867 podStartE2EDuration="1.779547867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:59:57.776216265 +0000 UTC m=+1970.395031179" watchObservedRunningTime="2026-09-15 22:59:57.779547867 +0000 UTC m=+1970.398362780"
Sep 15 23:00:00.130897 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.130856    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm"]
Sep 15 23:00:00.134693 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.134670    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm"
Sep 15 23:00:00.135368 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.135345    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825220-2t7mh"]
Sep 15 23:00:00.137384 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.137363    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 23:00:00.138559 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.138539    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-668e8d-29825220-hr2xr"]
Sep 15 23:00:00.138661 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.138645    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825220-2t7mh"
Sep 15 23:00:00.141890 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.141872    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-668e8d-29825220-hr2xr"
Sep 15 23:00:00.148165 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.148004    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm"
Sep 15 23:00:00.148454 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.148426    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm"]
Sep 15 23:00:00.150960 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.150938    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-668e8d-29825220-hr2xr"]
Sep 15 23:00:00.154638 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.153867    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825220-2t7mh"]
Sep 15 23:00:00.157400 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.157186    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-637a6f-29825220-5gkdk"]
Sep 15 23:00:00.162510 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.161730    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-637a6f-29825220-5gkdk"
Sep 15 23:00:00.167827 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.167769    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-637a6f-29825220-5gkdk"]
Sep 15 23:00:00.187034 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.187008    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825220-2t7mh"
Sep 15 23:00:00.189429 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.189404    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-668e8d-29825220-hr2xr"
Sep 15 23:00:00.201349 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.201326    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-637a6f-29825220-5gkdk"
Sep 15 23:00:00.436309 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.436190    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm"]
Sep 15 23:00:00.439371 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:00:00.439342    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7009be47_521c_45d4_b0c7_a9d9bbedf44b.slice/crio-7f65ac03965d2718dbd922f42036a67750bba1e48f5e4dd229544ce9328437c9 WatchSource:0}: Error finding container 7f65ac03965d2718dbd922f42036a67750bba1e48f5e4dd229544ce9328437c9: Status 404 returned error can't find the container with id 7f65ac03965d2718dbd922f42036a67750bba1e48f5e4dd229544ce9328437c9
Sep 15 23:00:00.510868 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.510839    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825220-2t7mh"]
Sep 15 23:00:00.513766 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:00:00.513682    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2210ea7_1cd5_454f_ba06_ff594a74adc1.slice/crio-64a2dee4dac6f445a6758d6a686b7a03af069f5c75ea51ac25bec65f146d4e98 WatchSource:0}: Error finding container 64a2dee4dac6f445a6758d6a686b7a03af069f5c75ea51ac25bec65f146d4e98: Status 404 returned error can't find the container with id 64a2dee4dac6f445a6758d6a686b7a03af069f5c75ea51ac25bec65f146d4e98
Sep 15 23:00:00.514780 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.514757    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-668e8d-29825220-hr2xr"]
Sep 15 23:00:00.517319 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.517294    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-637a6f-29825220-5gkdk"]
Sep 15 23:00:00.518415 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:00:00.518394    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f5de0e_4488_484c_babd_954570ce3b58.slice/crio-68b6e20b81568362590f5528e2e5594154eba7a4f88d22ad3d7473d47f130972 WatchSource:0}: Error finding container 68b6e20b81568362590f5528e2e5594154eba7a4f88d22ad3d7473d47f130972: Status 404 returned error can't find the container with id 68b6e20b81568362590f5528e2e5594154eba7a4f88d22ad3d7473d47f130972
Sep 15 23:00:00.519630 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:00:00.519612    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c775994_0209_4c09_b63b_dc84b0331359.slice/crio-a3dd487fcb144a66b80cceffa6e6b7e53a8ba6a60ef7a7b8363d5b034c690afa WatchSource:0}: Error finding container a3dd487fcb144a66b80cceffa6e6b7e53a8ba6a60ef7a7b8363d5b034c690afa: Status 404 returned error can't find the container with id a3dd487fcb144a66b80cceffa6e6b7e53a8ba6a60ef7a7b8363d5b034c690afa
Sep 15 23:00:00.764123 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.764083    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-668e8d-29825220-hr2xr" event={"ID":"1c775994-0209-4c09-b63b-dc84b0331359","Type":"ContainerStarted","Data":"a3dd487fcb144a66b80cceffa6e6b7e53a8ba6a60ef7a7b8363d5b034c690afa"}
Sep 15 23:00:00.765392 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.765367    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm" event={"ID":"7009be47-521c-45d4-b0c7-a9d9bbedf44b","Type":"ContainerStarted","Data":"7f65ac03965d2718dbd922f42036a67750bba1e48f5e4dd229544ce9328437c9"}
Sep 15 23:00:00.766557 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.766529    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-637a6f-29825220-5gkdk" event={"ID":"35f5de0e-4488-484c-babd-954570ce3b58","Type":"ContainerStarted","Data":"68b6e20b81568362590f5528e2e5594154eba7a4f88d22ad3d7473d47f130972"}
Sep 15 23:00:00.767978 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.767956    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825220-2t7mh" event={"ID":"a2210ea7-1cd5-454f-ba06-ff594a74adc1","Type":"ContainerStarted","Data":"4464b00e7279613c025e6eb7c6945d5abe6ee527f9e3137ba8bc995c5705ed18"}
Sep 15 23:00:00.768077 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.767997    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825220-2t7mh" event={"ID":"a2210ea7-1cd5-454f-ba06-ff594a74adc1","Type":"ContainerStarted","Data":"64a2dee4dac6f445a6758d6a686b7a03af069f5c75ea51ac25bec65f146d4e98"}
Sep 15 23:00:00.784045 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:00.783897    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825220-2t7mh" podStartSLOduration=0.783875908 podStartE2EDuration="783.875908ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 23:00:00.783332978 +0000 UTC m=+1973.402147887" watchObservedRunningTime="2026-09-15 23:00:00.783875908 +0000 UTC m=+1973.402690817"
Sep 15 23:00:01.425080 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:00:01.425041    2584 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2210ea7_1cd5_454f_ba06_ff594a74adc1.slice/crio-conmon-4464b00e7279613c025e6eb7c6945d5abe6ee527f9e3137ba8bc995c5705ed18.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7009be47_521c_45d4_b0c7_a9d9bbedf44b.slice/crio-conmon-fd4ae6ec4a80931bdb2ec7b925e958a947e68e62231fabf7e547d5a6049c1e34.scope\": RecentStats: unable to find data in memory cache]"
Sep 15 23:00:01.775568 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:01.775483    2584 generic.go:356] "Generic (PLEG): container finished" podID="1c775994-0209-4c09-b63b-dc84b0331359" containerID="d28602cf1fe6bf3b3c94ac2abddb0a0ace938cf41c23b61b9fa497ba1ea7f0b2" exitCode=0
Sep 15 23:00:01.775719 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:01.775570    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-668e8d-29825220-hr2xr" event={"ID":"1c775994-0209-4c09-b63b-dc84b0331359","Type":"ContainerDied","Data":"d28602cf1fe6bf3b3c94ac2abddb0a0ace938cf41c23b61b9fa497ba1ea7f0b2"}
Sep 15 23:00:01.777031 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:01.777006    2584 generic.go:356] "Generic (PLEG): container finished" podID="7009be47-521c-45d4-b0c7-a9d9bbedf44b" containerID="fd4ae6ec4a80931bdb2ec7b925e958a947e68e62231fabf7e547d5a6049c1e34" exitCode=7
Sep 15 23:00:01.777154 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:01.777071    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm" event={"ID":"7009be47-521c-45d4-b0c7-a9d9bbedf44b","Type":"ContainerDied","Data":"fd4ae6ec4a80931bdb2ec7b925e958a947e68e62231fabf7e547d5a6049c1e34"}
Sep 15 23:00:01.777242 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:01.777228    2584 scope.go:117] "RemoveContainer" containerID="fd4ae6ec4a80931bdb2ec7b925e958a947e68e62231fabf7e547d5a6049c1e34"
Sep 15 23:00:01.778744 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:01.778722    2584 generic.go:356] "Generic (PLEG): container finished" podID="35f5de0e-4488-484c-babd-954570ce3b58" containerID="c026772b2d5503c23f27c146f94badd0ebe6457452b7c98efa6a3df7bbaa4b14" exitCode=0
Sep 15 23:00:01.778842 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:01.778800    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-637a6f-29825220-5gkdk" event={"ID":"35f5de0e-4488-484c-babd-954570ce3b58","Type":"ContainerDied","Data":"c026772b2d5503c23f27c146f94badd0ebe6457452b7c98efa6a3df7bbaa4b14"}
Sep 15 23:00:01.780482 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:01.780462    2584 generic.go:356] "Generic (PLEG): container finished" podID="a2210ea7-1cd5-454f-ba06-ff594a74adc1" containerID="4464b00e7279613c025e6eb7c6945d5abe6ee527f9e3137ba8bc995c5705ed18" exitCode=0
Sep 15 23:00:01.780562 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:01.780521    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825220-2t7mh" event={"ID":"a2210ea7-1cd5-454f-ba06-ff594a74adc1","Type":"ContainerDied","Data":"4464b00e7279613c025e6eb7c6945d5abe6ee527f9e3137ba8bc995c5705ed18"}
Sep 15 23:00:02.169591 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:02.169550    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:00:02.790132 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:02.790048    2584 generic.go:356] "Generic (PLEG): container finished" podID="7009be47-521c-45d4-b0c7-a9d9bbedf44b" containerID="7f00b6de584e4926b0ed5ba1bcea6cfd42316f64851b8900170d16f12a6b2971" exitCode=7
Sep 15 23:00:02.790596 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:02.790138    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm" event={"ID":"7009be47-521c-45d4-b0c7-a9d9bbedf44b","Type":"ContainerDied","Data":"7f00b6de584e4926b0ed5ba1bcea6cfd42316f64851b8900170d16f12a6b2971"}
Sep 15 23:00:02.790596 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:02.790210    2584 scope.go:117] "RemoveContainer" containerID="fd4ae6ec4a80931bdb2ec7b925e958a947e68e62231fabf7e547d5a6049c1e34"
Sep 15 23:00:02.790596 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:02.790319    2584 scope.go:117] "RemoveContainer" containerID="7f00b6de584e4926b0ed5ba1bcea6cfd42316f64851b8900170d16f12a6b2971"
Sep 15 23:00:02.791139 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:00:02.790591    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm_odh-ai-gateway-infra(7009be47-521c-45d4-b0c7-a9d9bbedf44b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm" podUID="7009be47-521c-45d4-b0c7-a9d9bbedf44b"
Sep 15 23:00:02.949540 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:02.949511    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825220-2t7mh"
Sep 15 23:00:02.991094 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:02.991070    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-637a6f-29825220-5gkdk"
Sep 15 23:00:03.006871 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:03.006848    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-668e8d-29825220-hr2xr"
Sep 15 23:00:03.797364 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:03.797329    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-668e8d-29825220-hr2xr"
Sep 15 23:00:03.797780 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:03.797328    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-668e8d-29825220-hr2xr" event={"ID":"1c775994-0209-4c09-b63b-dc84b0331359","Type":"ContainerDied","Data":"a3dd487fcb144a66b80cceffa6e6b7e53a8ba6a60ef7a7b8363d5b034c690afa"}
Sep 15 23:00:03.797780 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:03.797496    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3dd487fcb144a66b80cceffa6e6b7e53a8ba6a60ef7a7b8363d5b034c690afa"
Sep 15 23:00:03.799176 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:03.799142    2584 scope.go:117] "RemoveContainer" containerID="7f00b6de584e4926b0ed5ba1bcea6cfd42316f64851b8900170d16f12a6b2971"
Sep 15 23:00:03.799445 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:00:03.799421    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm_odh-ai-gateway-infra(7009be47-521c-45d4-b0c7-a9d9bbedf44b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm" podUID="7009be47-521c-45d4-b0c7-a9d9bbedf44b"
Sep 15 23:00:03.800364 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:03.800343    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-637a6f-29825220-5gkdk"
Sep 15 23:00:03.800466 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:03.800388    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-637a6f-29825220-5gkdk" event={"ID":"35f5de0e-4488-484c-babd-954570ce3b58","Type":"ContainerDied","Data":"68b6e20b81568362590f5528e2e5594154eba7a4f88d22ad3d7473d47f130972"}
Sep 15 23:00:03.800466 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:03.800433    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68b6e20b81568362590f5528e2e5594154eba7a4f88d22ad3d7473d47f130972"
Sep 15 23:00:03.801832 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:03.801806    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825220-2t7mh"
Sep 15 23:00:03.801928 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:03.801806    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825220-2t7mh" event={"ID":"a2210ea7-1cd5-454f-ba06-ff594a74adc1","Type":"ContainerDied","Data":"64a2dee4dac6f445a6758d6a686b7a03af069f5c75ea51ac25bec65f146d4e98"}
Sep 15 23:00:03.801928 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:03.801918    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64a2dee4dac6f445a6758d6a686b7a03af069f5c75ea51ac25bec65f146d4e98"
Sep 15 23:00:08.753898 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:08.753861    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"
Sep 15 23:00:08.754362 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:08.753938    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"
Sep 15 23:00:08.816325 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:08.816296    2584 scope.go:117] "RemoveContainer" containerID="35efeef0846d16ba162786657bc7844f57b815e495ae4a7ca6805d5ae25741e1"
Sep 15 23:00:08.825216 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:08.825197    2584 scope.go:117] "RemoveContainer" containerID="8c5ca628adc7b6781c88d946ff3733a677a408dee2ebde23f5517acabfae7ff7"
Sep 15 23:00:08.833736 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:08.833716    2584 scope.go:117] "RemoveContainer" containerID="530b160b9073e40a883fb7fbd8dd7fa7b4ae643e7b68a6df031392e67ae8489e"
Sep 15 23:00:08.841347 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:08.841245    2584 scope.go:117] "RemoveContainer" containerID="9c82be68e66d9c4e4f12f93b935110eadd9cc32b5ea0ad5de09d60fb953ee301"
Sep 15 23:00:14.893220 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:14.893191    2584 scope.go:117] "RemoveContainer" containerID="7f00b6de584e4926b0ed5ba1bcea6cfd42316f64851b8900170d16f12a6b2971"
Sep 15 23:00:15.869960 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:15.869927    2584 generic.go:356] "Generic (PLEG): container finished" podID="7009be47-521c-45d4-b0c7-a9d9bbedf44b" containerID="9690ad433905029f494e9bb002b3fba1e645b847700343871a6bc1c3b21718a6" exitCode=0
Sep 15 23:00:15.870115 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:15.870012    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm" event={"ID":"7009be47-521c-45d4-b0c7-a9d9bbedf44b","Type":"ContainerDied","Data":"9690ad433905029f494e9bb002b3fba1e645b847700343871a6bc1c3b21718a6"}
Sep 15 23:00:15.870115 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:15.870101    2584 scope.go:117] "RemoveContainer" containerID="7f00b6de584e4926b0ed5ba1bcea6cfd42316f64851b8900170d16f12a6b2971"
Sep 15 23:00:16.910435 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:16.910403    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm"]
Sep 15 23:00:17.022292 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:17.022240    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm"
Sep 15 23:00:17.884637 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:17.884600    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm" event={"ID":"7009be47-521c-45d4-b0c7-a9d9bbedf44b","Type":"ContainerDied","Data":"7f65ac03965d2718dbd922f42036a67750bba1e48f5e4dd229544ce9328437c9"}
Sep 15 23:00:17.884831 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:17.884656    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f65ac03965d2718dbd922f42036a67750bba1e48f5e4dd229544ce9328437c9"
Sep 15 23:00:17.884831 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:17.884617    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm"
Sep 15 23:00:17.915195 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:17.915165    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm"]
Sep 15 23:00:17.925709 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:17.925685    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-10ec22-29825220-k5dkm"]
Sep 15 23:00:18.083249 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.083221    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:18.316859 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.316828    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:18.734427 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.734339    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h"]
Sep 15 23:00:18.734865 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.734846    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7009be47-521c-45d4-b0c7-a9d9bbedf44b" containerName="cleanup"
Sep 15 23:00:18.734961 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.734868    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7009be47-521c-45d4-b0c7-a9d9bbedf44b" containerName="cleanup"
Sep 15 23:00:18.734961 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.734883    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35f5de0e-4488-484c-babd-954570ce3b58" containerName="cleanup"
Sep 15 23:00:18.734961 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.734890    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="35f5de0e-4488-484c-babd-954570ce3b58" containerName="cleanup"
Sep 15 23:00:18.734961 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.734904    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7009be47-521c-45d4-b0c7-a9d9bbedf44b" containerName="cleanup"
Sep 15 23:00:18.734961 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.734912    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7009be47-521c-45d4-b0c7-a9d9bbedf44b" containerName="cleanup"
Sep 15 23:00:18.734961 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.734947    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c775994-0209-4c09-b63b-dc84b0331359" containerName="cleanup"
Sep 15 23:00:18.734961 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.734955    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c775994-0209-4c09-b63b-dc84b0331359" containerName="cleanup"
Sep 15 23:00:18.735339 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.734967    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7009be47-521c-45d4-b0c7-a9d9bbedf44b" containerName="cleanup"
Sep 15 23:00:18.735339 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.734975    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7009be47-521c-45d4-b0c7-a9d9bbedf44b" containerName="cleanup"
Sep 15 23:00:18.735339 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.734993    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2210ea7-1cd5-454f-ba06-ff594a74adc1" containerName="cleanup"
Sep 15 23:00:18.735339 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.735001    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2210ea7-1cd5-454f-ba06-ff594a74adc1" containerName="cleanup"
Sep 15 23:00:18.735339 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.735106    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="35f5de0e-4488-484c-babd-954570ce3b58" containerName="cleanup"
Sep 15 23:00:18.735339 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.735120    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7009be47-521c-45d4-b0c7-a9d9bbedf44b" containerName="cleanup"
Sep 15 23:00:18.735339 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.735129    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2210ea7-1cd5-454f-ba06-ff594a74adc1" containerName="cleanup"
Sep 15 23:00:18.735339 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.735142    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c775994-0209-4c09-b63b-dc84b0331359" containerName="cleanup"
Sep 15 23:00:18.735339 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.735154    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7009be47-521c-45d4-b0c7-a9d9bbedf44b" containerName="cleanup"
Sep 15 23:00:18.738495 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.738474    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h"
Sep 15 23:00:18.741071 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.741048    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 23:00:18.741174 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.741097    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 23:00:18.746845 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.746821    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h"]
Sep 15 23:00:18.856831 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.856790    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c199204f-fdfa-480b-b2be-59795bdf3b1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h\" (UID: \"c199204f-fdfa-480b-b2be-59795bdf3b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h"
Sep 15 23:00:18.958144 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.958096    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c199204f-fdfa-480b-b2be-59795bdf3b1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h\" (UID: \"c199204f-fdfa-480b-b2be-59795bdf3b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h"
Sep 15 23:00:18.958751 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:18.958729    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c199204f-fdfa-480b-b2be-59795bdf3b1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h\" (UID: \"c199204f-fdfa-480b-b2be-59795bdf3b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h"
Sep 15 23:00:19.050362 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:19.050334    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h"
Sep 15 23:00:19.331701 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:19.324038    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h"]
Sep 15 23:00:19.334170 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:00:19.334136    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc199204f_fdfa_480b_b2be_59795bdf3b1b.slice/crio-d447bb29890e6cf600dfefc5f891817155c911cad67ebed67b0a52a5a450bafd WatchSource:0}: Error finding container d447bb29890e6cf600dfefc5f891817155c911cad67ebed67b0a52a5a450bafd: Status 404 returned error can't find the container with id d447bb29890e6cf600dfefc5f891817155c911cad67ebed67b0a52a5a450bafd
Sep 15 23:00:19.341431 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:19.341400    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:19.899171 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:19.899088    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7009be47-521c-45d4-b0c7-a9d9bbedf44b" path="/var/lib/kubelet/pods/7009be47-521c-45d4-b0c7-a9d9bbedf44b/volumes"
Sep 15 23:00:19.899386 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:19.899368    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h" event={"ID":"c199204f-fdfa-480b-b2be-59795bdf3b1b","Type":"ContainerStarted","Data":"6906d2759794e309154961980ee9dd86d8abd22948f8fe6b65e93226df56da50"}
Sep 15 23:00:19.899437 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:19.899420    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h" event={"ID":"c199204f-fdfa-480b-b2be-59795bdf3b1b","Type":"ContainerStarted","Data":"d447bb29890e6cf600dfefc5f891817155c911cad67ebed67b0a52a5a450bafd"}
Sep 15 23:00:19.926726 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:19.926681    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h" podStartSLOduration=1.9266605540000001 podStartE2EDuration="1.926660554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 23:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 23:00:19.92534865 +0000 UTC m=+1992.544163559" watchObservedRunningTime="2026-09-15 23:00:19.926660554 +0000 UTC m=+1992.545475466"
Sep 15 23:00:20.154209 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:20.154118    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:20.855971 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:20.855942    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:20.903745 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:20.903709    2584 generic.go:356] "Generic (PLEG): container finished" podID="c199204f-fdfa-480b-b2be-59795bdf3b1b" containerID="6906d2759794e309154961980ee9dd86d8abd22948f8fe6b65e93226df56da50" exitCode=0
Sep 15 23:00:20.903960 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:20.903753    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h" event={"ID":"c199204f-fdfa-480b-b2be-59795bdf3b1b","Type":"ContainerDied","Data":"6906d2759794e309154961980ee9dd86d8abd22948f8fe6b65e93226df56da50"}
Sep 15 23:00:21.675763 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:21.675724    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:22.045568 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:22.045546    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h"
Sep 15 23:00:22.188846 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:22.188810    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c199204f-fdfa-480b-b2be-59795bdf3b1b-maas-api-service-ca\") pod \"c199204f-fdfa-480b-b2be-59795bdf3b1b\" (UID: \"c199204f-fdfa-480b-b2be-59795bdf3b1b\") "
Sep 15 23:00:22.189226 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:22.189204    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c199204f-fdfa-480b-b2be-59795bdf3b1b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c199204f-fdfa-480b-b2be-59795bdf3b1b" (UID: "c199204f-fdfa-480b-b2be-59795bdf3b1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 23:00:22.289860 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:22.289776    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c199204f-fdfa-480b-b2be-59795bdf3b1b-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:00:22.464462 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:22.464429    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:22.915292 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:22.915248    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h"
Sep 15 23:00:22.915690 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:22.915292    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h" event={"ID":"c199204f-fdfa-480b-b2be-59795bdf3b1b","Type":"ContainerDied","Data":"d447bb29890e6cf600dfefc5f891817155c911cad67ebed67b0a52a5a450bafd"}
Sep 15 23:00:22.915690 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:22.915336    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d447bb29890e6cf600dfefc5f891817155c911cad67ebed67b0a52a5a450bafd"
Sep 15 23:00:23.279784 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:23.279677    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:24.675115 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:24.675074    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:25.295607 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.295574    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"]
Sep 15 23:00:25.296156 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.296117    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l" podUID="1be39ce3-ffd0-4d25-bf14-2c29292d5459" containerName="payload-processing" containerID="cri-o://9922ee812f5650bcd7458ae10a69373ded94f9336df3c41b279c19201484df79" gracePeriod=30
Sep 15 23:00:25.304386 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.304355    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"]
Sep 15 23:00:25.304768 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.304554    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b" podUID="b878802c-c763-41a0-be2e-e5365010b5cb" containerName="payload-pre-processing" containerID="cri-o://16daf934e8a355d651d5a5b61b11d1a60f7f0a89a349ca816c8759da2a18e09b" gracePeriod=30
Sep 15 23:00:25.331317 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.331287    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:25.429125 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.429083    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:25.464597 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.464545    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h"]
Sep 15 23:00:25.469918 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.469864    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-10ec22-mxd4h"]
Sep 15 23:00:25.532746 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.532712    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:25.608379 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.608353    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"
Sep 15 23:00:25.611939 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.611915    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"
Sep 15 23:00:25.721094 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.721061    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be39ce3-ffd0-4d25-bf14-2c29292d5459-plugins-config-volume\") pod \"1be39ce3-ffd0-4d25-bf14-2c29292d5459\" (UID: \"1be39ce3-ffd0-4d25-bf14-2c29292d5459\") "
Sep 15 23:00:25.721523 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.721109    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xmpg\" (UniqueName: \"kubernetes.io/projected/b878802c-c763-41a0-be2e-e5365010b5cb-kube-api-access-5xmpg\") pod \"b878802c-c763-41a0-be2e-e5365010b5cb\" (UID: \"b878802c-c763-41a0-be2e-e5365010b5cb\") "
Sep 15 23:00:25.721523 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.721166    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vmq2\" (UniqueName: \"kubernetes.io/projected/1be39ce3-ffd0-4d25-bf14-2c29292d5459-kube-api-access-4vmq2\") pod \"1be39ce3-ffd0-4d25-bf14-2c29292d5459\" (UID: \"1be39ce3-ffd0-4d25-bf14-2c29292d5459\") "
Sep 15 23:00:25.721523 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.721241    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b878802c-c763-41a0-be2e-e5365010b5cb-plugins-config-volume\") pod \"b878802c-c763-41a0-be2e-e5365010b5cb\" (UID: \"b878802c-c763-41a0-be2e-e5365010b5cb\") "
Sep 15 23:00:25.721704 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.721519    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be39ce3-ffd0-4d25-bf14-2c29292d5459-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1be39ce3-ffd0-4d25-bf14-2c29292d5459" (UID: "1be39ce3-ffd0-4d25-bf14-2c29292d5459"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 23:00:25.721767 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.721721    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b878802c-c763-41a0-be2e-e5365010b5cb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b878802c-c763-41a0-be2e-e5365010b5cb" (UID: "b878802c-c763-41a0-be2e-e5365010b5cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 23:00:25.723206 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.723182    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b878802c-c763-41a0-be2e-e5365010b5cb-kube-api-access-5xmpg" (OuterVolumeSpecName: "kube-api-access-5xmpg") pod "b878802c-c763-41a0-be2e-e5365010b5cb" (UID: "b878802c-c763-41a0-be2e-e5365010b5cb"). InnerVolumeSpecName "kube-api-access-5xmpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 23:00:25.723326 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.723301    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be39ce3-ffd0-4d25-bf14-2c29292d5459-kube-api-access-4vmq2" (OuterVolumeSpecName: "kube-api-access-4vmq2") pod "1be39ce3-ffd0-4d25-bf14-2c29292d5459" (UID: "1be39ce3-ffd0-4d25-bf14-2c29292d5459"). InnerVolumeSpecName "kube-api-access-4vmq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 23:00:25.818590 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.818554    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:25.822786 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.822762    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b878802c-c763-41a0-be2e-e5365010b5cb-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:00:25.822878 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.822787    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be39ce3-ffd0-4d25-bf14-2c29292d5459-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:00:25.822878 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.822800    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xmpg\" (UniqueName: \"kubernetes.io/projected/b878802c-c763-41a0-be2e-e5365010b5cb-kube-api-access-5xmpg\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:00:25.822878 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.822809    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vmq2\" (UniqueName: \"kubernetes.io/projected/1be39ce3-ffd0-4d25-bf14-2c29292d5459-kube-api-access-4vmq2\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:00:25.899601 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.899519    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c199204f-fdfa-480b-b2be-59795bdf3b1b" path="/var/lib/kubelet/pods/c199204f-fdfa-480b-b2be-59795bdf3b1b/volumes"
Sep 15 23:00:25.927010 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.926979    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:25.932405 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.932383    2584 generic.go:356] "Generic (PLEG): container finished" podID="b878802c-c763-41a0-be2e-e5365010b5cb" containerID="16daf934e8a355d651d5a5b61b11d1a60f7f0a89a349ca816c8759da2a18e09b" exitCode=0
Sep 15 23:00:25.932527 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.932444    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"
Sep 15 23:00:25.932527 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.932470    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b" event={"ID":"b878802c-c763-41a0-be2e-e5365010b5cb","Type":"ContainerDied","Data":"16daf934e8a355d651d5a5b61b11d1a60f7f0a89a349ca816c8759da2a18e09b"}
Sep 15 23:00:25.932636 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.932529    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b" event={"ID":"b878802c-c763-41a0-be2e-e5365010b5cb","Type":"ContainerDied","Data":"71c65ce72e63ee768cda81db6e8a99d8f90a6d05c7cf544f4d3c590f7c40e828"}
Sep 15 23:00:25.932636 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.932562    2584 scope.go:117] "RemoveContainer" containerID="16daf934e8a355d651d5a5b61b11d1a60f7f0a89a349ca816c8759da2a18e09b"
Sep 15 23:00:25.933948 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.933925    2584 generic.go:356] "Generic (PLEG): container finished" podID="1be39ce3-ffd0-4d25-bf14-2c29292d5459" containerID="9922ee812f5650bcd7458ae10a69373ded94f9336df3c41b279c19201484df79" exitCode=0
Sep 15 23:00:25.934052 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.934001    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"
Sep 15 23:00:25.934052 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.934038    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l" event={"ID":"1be39ce3-ffd0-4d25-bf14-2c29292d5459","Type":"ContainerDied","Data":"9922ee812f5650bcd7458ae10a69373ded94f9336df3c41b279c19201484df79"}
Sep 15 23:00:25.934166 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.934074    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l" event={"ID":"1be39ce3-ffd0-4d25-bf14-2c29292d5459","Type":"ContainerDied","Data":"c3c31c9ba433a24e096a0a5e922a76b34ba4d36b661d185da5fda8232aba042d"}
Sep 15 23:00:25.941838 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.941806    2584 scope.go:117] "RemoveContainer" containerID="16daf934e8a355d651d5a5b61b11d1a60f7f0a89a349ca816c8759da2a18e09b"
Sep 15 23:00:25.942116 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:00:25.942090    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16daf934e8a355d651d5a5b61b11d1a60f7f0a89a349ca816c8759da2a18e09b\": container with ID starting with 16daf934e8a355d651d5a5b61b11d1a60f7f0a89a349ca816c8759da2a18e09b not found: ID does not exist" containerID="16daf934e8a355d651d5a5b61b11d1a60f7f0a89a349ca816c8759da2a18e09b"
Sep 15 23:00:25.942116 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.942116    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16daf934e8a355d651d5a5b61b11d1a60f7f0a89a349ca816c8759da2a18e09b"} err="failed to get container status \"16daf934e8a355d651d5a5b61b11d1a60f7f0a89a349ca816c8759da2a18e09b\": rpc error: code = NotFound desc = could not find container \"16daf934e8a355d651d5a5b61b11d1a60f7f0a89a349ca816c8759da2a18e09b\": container with ID starting with 16daf934e8a355d651d5a5b61b11d1a60f7f0a89a349ca816c8759da2a18e09b not found: ID does not exist"
Sep 15 23:00:25.942116 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.942134    2584 scope.go:117] "RemoveContainer" containerID="9922ee812f5650bcd7458ae10a69373ded94f9336df3c41b279c19201484df79"
Sep 15 23:00:25.950387 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.950372    2584 scope.go:117] "RemoveContainer" containerID="9922ee812f5650bcd7458ae10a69373ded94f9336df3c41b279c19201484df79"
Sep 15 23:00:25.950652 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:00:25.950633    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9922ee812f5650bcd7458ae10a69373ded94f9336df3c41b279c19201484df79\": container with ID starting with 9922ee812f5650bcd7458ae10a69373ded94f9336df3c41b279c19201484df79 not found: ID does not exist" containerID="9922ee812f5650bcd7458ae10a69373ded94f9336df3c41b279c19201484df79"
Sep 15 23:00:25.950726 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.950660    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9922ee812f5650bcd7458ae10a69373ded94f9336df3c41b279c19201484df79"} err="failed to get container status \"9922ee812f5650bcd7458ae10a69373ded94f9336df3c41b279c19201484df79\": rpc error: code = NotFound desc = could not find container \"9922ee812f5650bcd7458ae10a69373ded94f9336df3c41b279c19201484df79\": container with ID starting with 9922ee812f5650bcd7458ae10a69373ded94f9336df3c41b279c19201484df79 not found: ID does not exist"
Sep 15 23:00:25.953633 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.953612    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"]
Sep 15 23:00:25.957288 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.957248    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-10ec22-5f88679546-zqf5b"]
Sep 15 23:00:25.967789 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.967769    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"]
Sep 15 23:00:25.969727 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:25.969704    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-10ec22-5d79dfd9cc-dtl2l"]
Sep 15 23:00:26.616527 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:26.616476    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:27.466529 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:27.466498    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:27.898952 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:27.898918    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1be39ce3-ffd0-4d25-bf14-2c29292d5459" path="/var/lib/kubelet/pods/1be39ce3-ffd0-4d25-bf14-2c29292d5459/volumes"
Sep 15 23:00:27.899329 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:27.899316    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b878802c-c763-41a0-be2e-e5365010b5cb" path="/var/lib/kubelet/pods/b878802c-c763-41a0-be2e-e5365010b5cb/volumes"
Sep 15 23:00:28.153132 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:28.153018    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:29.092510 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:29.092462    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:30.263083 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:30.263042    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:00:31.632465 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:31.632429    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:31.698697 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:31.698665    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:31.884426 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:31.884350    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:35.393144 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.393106    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px"]
Sep 15 23:00:35.393807 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.393783    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c199204f-fdfa-480b-b2be-59795bdf3b1b" containerName="revoke-keys"
Sep 15 23:00:35.393867 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.393811    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c199204f-fdfa-480b-b2be-59795bdf3b1b" containerName="revoke-keys"
Sep 15 23:00:35.393867 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.393837    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b878802c-c763-41a0-be2e-e5365010b5cb" containerName="payload-pre-processing"
Sep 15 23:00:35.393867 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.393846    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b878802c-c763-41a0-be2e-e5365010b5cb" containerName="payload-pre-processing"
Sep 15 23:00:35.393959 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.393865    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1be39ce3-ffd0-4d25-bf14-2c29292d5459" containerName="payload-processing"
Sep 15 23:00:35.393959 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.393875    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1be39ce3-ffd0-4d25-bf14-2c29292d5459" containerName="payload-processing"
Sep 15 23:00:35.394020 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.393973    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b878802c-c763-41a0-be2e-e5365010b5cb" containerName="payload-pre-processing"
Sep 15 23:00:35.394020 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.393988    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c199204f-fdfa-480b-b2be-59795bdf3b1b" containerName="revoke-keys"
Sep 15 23:00:35.394020 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.393999    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1be39ce3-ffd0-4d25-bf14-2c29292d5459" containerName="payload-processing"
Sep 15 23:00:35.394020 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.394008    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7009be47-521c-45d4-b0c7-a9d9bbedf44b" containerName="cleanup"
Sep 15 23:00:35.398707 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.398688    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px"
Sep 15 23:00:35.402369 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.402344    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 23:00:35.402688 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.402369    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 23:00:35.404378 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.404355    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px"]
Sep 15 23:00:35.518556 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.518518    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a9600-bed1-4b40-b7bf-d9c17e8f17cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px\" (UID: \"d32a9600-bed1-4b40-b7bf-d9c17e8f17cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px"
Sep 15 23:00:35.619394 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.619357    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a9600-bed1-4b40-b7bf-d9c17e8f17cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px\" (UID: \"d32a9600-bed1-4b40-b7bf-d9c17e8f17cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px"
Sep 15 23:00:35.620176 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.620151    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a9600-bed1-4b40-b7bf-d9c17e8f17cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px\" (UID: \"d32a9600-bed1-4b40-b7bf-d9c17e8f17cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px"
Sep 15 23:00:35.720442 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.720340    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px"
Sep 15 23:00:35.991659 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:35.991630    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px"]
Sep 15 23:00:35.993362 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:00:35.993328    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32a9600_bed1_4b40_b7bf_d9c17e8f17cb.slice/crio-3efb266d9c9c2122c547a02404bc081f6ad66c0297aed2b6ef6d82b816a81751 WatchSource:0}: Error finding container 3efb266d9c9c2122c547a02404bc081f6ad66c0297aed2b6ef6d82b816a81751: Status 404 returned error can't find the container with id 3efb266d9c9c2122c547a02404bc081f6ad66c0297aed2b6ef6d82b816a81751
Sep 15 23:00:36.034193 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:36.034165    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:36.357802 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:36.357760    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:36.994520 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:36.994421    2584 generic.go:356] "Generic (PLEG): container finished" podID="d32a9600-bed1-4b40-b7bf-d9c17e8f17cb" containerID="7de742a0cbfcce69c32035ed1ca921ed7b250f6aba63a09bdaee6fec89c696f6" exitCode=0
Sep 15 23:00:36.994873 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:36.994514    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px" event={"ID":"d32a9600-bed1-4b40-b7bf-d9c17e8f17cb","Type":"ContainerDied","Data":"7de742a0cbfcce69c32035ed1ca921ed7b250f6aba63a09bdaee6fec89c696f6"}
Sep 15 23:00:36.994873 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:36.994578    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px" event={"ID":"d32a9600-bed1-4b40-b7bf-d9c17e8f17cb","Type":"ContainerStarted","Data":"3efb266d9c9c2122c547a02404bc081f6ad66c0297aed2b6ef6d82b816a81751"}
Sep 15 23:00:37.156360 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:37.156330    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:37.964208 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:37.964165    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:38.142099 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:38.142074    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px"
Sep 15 23:00:38.245317 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:38.245210    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a9600-bed1-4b40-b7bf-d9c17e8f17cb-maas-api-service-ca\") pod \"d32a9600-bed1-4b40-b7bf-d9c17e8f17cb\" (UID: \"d32a9600-bed1-4b40-b7bf-d9c17e8f17cb\") "
Sep 15 23:00:38.245612 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:38.245589    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32a9600-bed1-4b40-b7bf-d9c17e8f17cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d32a9600-bed1-4b40-b7bf-d9c17e8f17cb" (UID: "d32a9600-bed1-4b40-b7bf-d9c17e8f17cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 23:00:38.346092 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:38.346056    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a9600-bed1-4b40-b7bf-d9c17e8f17cb-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:00:38.696353 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:38.696315    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:39.006848 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:39.006757    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px"
Sep 15 23:00:39.006848 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:39.006796    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px" event={"ID":"d32a9600-bed1-4b40-b7bf-d9c17e8f17cb","Type":"ContainerDied","Data":"3efb266d9c9c2122c547a02404bc081f6ad66c0297aed2b6ef6d82b816a81751"}
Sep 15 23:00:39.007037 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:39.006851    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3efb266d9c9c2122c547a02404bc081f6ad66c0297aed2b6ef6d82b816a81751"
Sep 15 23:00:39.658031 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:39.657998    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:48.578247 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:48.578202    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:00:49.226253 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:49.226217    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:51.060457 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.060423    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-637a6f-29825220-5gkdk"]
Sep 15 23:00:51.062705 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.062676    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:51.063641 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.063621    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-637a6f-29825220-5gkdk"]
Sep 15 23:00:51.085351 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.085326    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"]
Sep 15 23:00:51.085565 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.085520    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx" podUID="fe778af7-75f2-44ec-aa47-c089df42f5d5" containerName="payload-processing" containerID="cri-o://24c9546b9fd52dd2ba87b9b48151553c2308ae4ceb6299513a1445ebe4ec41fd" gracePeriod=30
Sep 15 23:00:51.095871 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.095837    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"]
Sep 15 23:00:51.096337 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.096079    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p" podUID="9d2269c2-3c50-4443-9736-ba6187567366" containerName="payload-pre-processing" containerID="cri-o://d58d9efbc482911f41ef8ecf310c78e4c0586b7c904e57f8ce308b986cc5f830" gracePeriod=30
Sep 15 23:00:51.133367 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.133337    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:51.212817 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.212783    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:51.270015 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.269983    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px"]
Sep 15 23:00:51.272432 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.272404    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-637a6f-sz9px"]
Sep 15 23:00:51.279175 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.279143    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:51.382800 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.382778    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"
Sep 15 23:00:51.385958 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.385937    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"
Sep 15 23:00:51.464751 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.464722    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tgrx\" (UniqueName: \"kubernetes.io/projected/fe778af7-75f2-44ec-aa47-c089df42f5d5-kube-api-access-6tgrx\") pod \"fe778af7-75f2-44ec-aa47-c089df42f5d5\" (UID: \"fe778af7-75f2-44ec-aa47-c089df42f5d5\") "
Sep 15 23:00:51.464915 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.464762    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe778af7-75f2-44ec-aa47-c089df42f5d5-plugins-config-volume\") pod \"fe778af7-75f2-44ec-aa47-c089df42f5d5\" (UID: \"fe778af7-75f2-44ec-aa47-c089df42f5d5\") "
Sep 15 23:00:51.464915 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.464820    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2269c2-3c50-4443-9736-ba6187567366-plugins-config-volume\") pod \"9d2269c2-3c50-4443-9736-ba6187567366\" (UID: \"9d2269c2-3c50-4443-9736-ba6187567366\") "
Sep 15 23:00:51.464915 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.464872    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx4qw\" (UniqueName: \"kubernetes.io/projected/9d2269c2-3c50-4443-9736-ba6187567366-kube-api-access-kx4qw\") pod \"9d2269c2-3c50-4443-9736-ba6187567366\" (UID: \"9d2269c2-3c50-4443-9736-ba6187567366\") "
Sep 15 23:00:51.465192 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.465152    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe778af7-75f2-44ec-aa47-c089df42f5d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe778af7-75f2-44ec-aa47-c089df42f5d5" (UID: "fe778af7-75f2-44ec-aa47-c089df42f5d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 23:00:51.465192 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.465160    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d2269c2-3c50-4443-9736-ba6187567366-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d2269c2-3c50-4443-9736-ba6187567366" (UID: "9d2269c2-3c50-4443-9736-ba6187567366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 23:00:51.466903 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.466875    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe778af7-75f2-44ec-aa47-c089df42f5d5-kube-api-access-6tgrx" (OuterVolumeSpecName: "kube-api-access-6tgrx") pod "fe778af7-75f2-44ec-aa47-c089df42f5d5" (UID: "fe778af7-75f2-44ec-aa47-c089df42f5d5"). InnerVolumeSpecName "kube-api-access-6tgrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 23:00:51.466999 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.466953    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2269c2-3c50-4443-9736-ba6187567366-kube-api-access-kx4qw" (OuterVolumeSpecName: "kube-api-access-kx4qw") pod "9d2269c2-3c50-4443-9736-ba6187567366" (UID: "9d2269c2-3c50-4443-9736-ba6187567366"). InnerVolumeSpecName "kube-api-access-kx4qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 23:00:51.566619 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.566587    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx4qw\" (UniqueName: \"kubernetes.io/projected/9d2269c2-3c50-4443-9736-ba6187567366-kube-api-access-kx4qw\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:00:51.566619 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.566613    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tgrx\" (UniqueName: \"kubernetes.io/projected/fe778af7-75f2-44ec-aa47-c089df42f5d5-kube-api-access-6tgrx\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:00:51.566619 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.566622    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe778af7-75f2-44ec-aa47-c089df42f5d5-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:00:51.566844 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.566631    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2269c2-3c50-4443-9736-ba6187567366-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:00:51.627203 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.627119    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:51.745564 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.745520    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:51.899430 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.899349    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35f5de0e-4488-484c-babd-954570ce3b58" path="/var/lib/kubelet/pods/35f5de0e-4488-484c-babd-954570ce3b58/volumes"
Sep 15 23:00:51.899697 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:51.899683    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d32a9600-bed1-4b40-b7bf-d9c17e8f17cb" path="/var/lib/kubelet/pods/d32a9600-bed1-4b40-b7bf-d9c17e8f17cb/volumes"
Sep 15 23:00:52.076716 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.076684    2584 generic.go:356] "Generic (PLEG): container finished" podID="9d2269c2-3c50-4443-9736-ba6187567366" containerID="d58d9efbc482911f41ef8ecf310c78e4c0586b7c904e57f8ce308b986cc5f830" exitCode=0
Sep 15 23:00:52.077124 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.076730    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"
Sep 15 23:00:52.077124 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.076767    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p" event={"ID":"9d2269c2-3c50-4443-9736-ba6187567366","Type":"ContainerDied","Data":"d58d9efbc482911f41ef8ecf310c78e4c0586b7c904e57f8ce308b986cc5f830"}
Sep 15 23:00:52.077124 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.076825    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p" event={"ID":"9d2269c2-3c50-4443-9736-ba6187567366","Type":"ContainerDied","Data":"ca08027d87d405e8360dce3e155ab250258eb301486bfd1d124c5261100bf4c0"}
Sep 15 23:00:52.077124 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.076860    2584 scope.go:117] "RemoveContainer" containerID="d58d9efbc482911f41ef8ecf310c78e4c0586b7c904e57f8ce308b986cc5f830"
Sep 15 23:00:52.078179 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.078158    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe778af7-75f2-44ec-aa47-c089df42f5d5" containerID="24c9546b9fd52dd2ba87b9b48151553c2308ae4ceb6299513a1445ebe4ec41fd" exitCode=0
Sep 15 23:00:52.078305 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.078188    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx" event={"ID":"fe778af7-75f2-44ec-aa47-c089df42f5d5","Type":"ContainerDied","Data":"24c9546b9fd52dd2ba87b9b48151553c2308ae4ceb6299513a1445ebe4ec41fd"}
Sep 15 23:00:52.078305 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.078206    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"
Sep 15 23:00:52.078305 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.078220    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx" event={"ID":"fe778af7-75f2-44ec-aa47-c089df42f5d5","Type":"ContainerDied","Data":"fefb65bf6d81acddd7ad0df03884b6696c34fd0b2028855943c64e941994a50f"}
Sep 15 23:00:52.086311 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.086081    2584 scope.go:117] "RemoveContainer" containerID="d58d9efbc482911f41ef8ecf310c78e4c0586b7c904e57f8ce308b986cc5f830"
Sep 15 23:00:52.086426 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:00:52.086407    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58d9efbc482911f41ef8ecf310c78e4c0586b7c904e57f8ce308b986cc5f830\": container with ID starting with d58d9efbc482911f41ef8ecf310c78e4c0586b7c904e57f8ce308b986cc5f830 not found: ID does not exist" containerID="d58d9efbc482911f41ef8ecf310c78e4c0586b7c904e57f8ce308b986cc5f830"
Sep 15 23:00:52.086495 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.086433    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58d9efbc482911f41ef8ecf310c78e4c0586b7c904e57f8ce308b986cc5f830"} err="failed to get container status \"d58d9efbc482911f41ef8ecf310c78e4c0586b7c904e57f8ce308b986cc5f830\": rpc error: code = NotFound desc = could not find container \"d58d9efbc482911f41ef8ecf310c78e4c0586b7c904e57f8ce308b986cc5f830\": container with ID starting with d58d9efbc482911f41ef8ecf310c78e4c0586b7c904e57f8ce308b986cc5f830 not found: ID does not exist"
Sep 15 23:00:52.086495 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.086452    2584 scope.go:117] "RemoveContainer" containerID="24c9546b9fd52dd2ba87b9b48151553c2308ae4ceb6299513a1445ebe4ec41fd"
Sep 15 23:00:52.094551 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.094442    2584 scope.go:117] "RemoveContainer" containerID="24c9546b9fd52dd2ba87b9b48151553c2308ae4ceb6299513a1445ebe4ec41fd"
Sep 15 23:00:52.094900 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:00:52.094879    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c9546b9fd52dd2ba87b9b48151553c2308ae4ceb6299513a1445ebe4ec41fd\": container with ID starting with 24c9546b9fd52dd2ba87b9b48151553c2308ae4ceb6299513a1445ebe4ec41fd not found: ID does not exist" containerID="24c9546b9fd52dd2ba87b9b48151553c2308ae4ceb6299513a1445ebe4ec41fd"
Sep 15 23:00:52.094988 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.094906    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c9546b9fd52dd2ba87b9b48151553c2308ae4ceb6299513a1445ebe4ec41fd"} err="failed to get container status \"24c9546b9fd52dd2ba87b9b48151553c2308ae4ceb6299513a1445ebe4ec41fd\": rpc error: code = NotFound desc = could not find container \"24c9546b9fd52dd2ba87b9b48151553c2308ae4ceb6299513a1445ebe4ec41fd\": container with ID starting with 24c9546b9fd52dd2ba87b9b48151553c2308ae4ceb6299513a1445ebe4ec41fd not found: ID does not exist"
Sep 15 23:00:52.097067 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.097047    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"]
Sep 15 23:00:52.101438 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.101417    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-637a6f-7bf8b8c98c-qh6jx"]
Sep 15 23:00:52.112152 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.112131    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"]
Sep 15 23:00:52.117980 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.117960    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-637a6f-78755d944b-wmn4p"]
Sep 15 23:00:52.453135 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:52.453100    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:53.388460 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:53.388418    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:53.899325 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:53.899285    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d2269c2-3c50-4443-9736-ba6187567366" path="/var/lib/kubelet/pods/9d2269c2-3c50-4443-9736-ba6187567366/volumes"
Sep 15 23:00:53.899696 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:53.899682    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe778af7-75f2-44ec-aa47-c089df42f5d5" path="/var/lib/kubelet/pods/fe778af7-75f2-44ec-aa47-c089df42f5d5/volumes"
Sep 15 23:00:54.184160 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:54.184067    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:00:56.698960 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:00:56.698929    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:01.856331 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:01.856295    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:01:08.875134 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:08.875104    2584 scope.go:117] "RemoveContainer" containerID="19eb4411609b6b0af77a0f88de53116f990071237bf5b974ebdd5ac57113bd34"
Sep 15 23:01:08.883737 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:08.883714    2584 scope.go:117] "RemoveContainer" containerID="2540a5f4f4a411e4592b68a866021c1e0f7140adc36025e283a97882389368ec"
Sep 15 23:01:32.900574 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:32.900491    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:33.670434 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:33.670395    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:39.232599 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:39.232548    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:44.785821 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:44.785790    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:44.848177 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:44.848151    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:45.072811 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.072769    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"]
Sep 15 23:01:45.073011 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.072984    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" podUID="51454f48-4753-4ebf-9979-26f01e42e967" containerName="istio-proxy" containerID="cri-o://31636ee71f25902d9ee50cef7188ae8fbcbd386e7b707c0af2cf178c12c7cc47" gracePeriod=30
Sep 15 23:01:45.103422 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.103389    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:45.487638 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.487549    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr"]
Sep 15 23:01:45.488173 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.488150    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe778af7-75f2-44ec-aa47-c089df42f5d5" containerName="payload-processing"
Sep 15 23:01:45.488321 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.488174    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe778af7-75f2-44ec-aa47-c089df42f5d5" containerName="payload-processing"
Sep 15 23:01:45.488321 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.488190    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d2269c2-3c50-4443-9736-ba6187567366" containerName="payload-pre-processing"
Sep 15 23:01:45.488321 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.488199    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d2269c2-3c50-4443-9736-ba6187567366" containerName="payload-pre-processing"
Sep 15 23:01:45.488321 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.488218    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32a9600-bed1-4b40-b7bf-d9c17e8f17cb" containerName="revoke-keys"
Sep 15 23:01:45.488321 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.488227    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32a9600-bed1-4b40-b7bf-d9c17e8f17cb" containerName="revoke-keys"
Sep 15 23:01:45.488598 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.488584    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d2269c2-3c50-4443-9736-ba6187567366" containerName="payload-pre-processing"
Sep 15 23:01:45.488654 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.488604    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d32a9600-bed1-4b40-b7bf-d9c17e8f17cb" containerName="revoke-keys"
Sep 15 23:01:45.488654 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.488618    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe778af7-75f2-44ec-aa47-c089df42f5d5" containerName="payload-processing"
Sep 15 23:01:45.492140 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.492119    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr"
Sep 15 23:01:45.495089 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.495062    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 23:01:45.496053 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.495868    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 23:01:45.498965 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.498942    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr"]
Sep 15 23:01:45.641487 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.641458    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c2aa8c-f081-47a8-aec2-eb84c7c9dda7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr\" (UID: \"38c2aa8c-f081-47a8-aec2-eb84c7c9dda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr"
Sep 15 23:01:45.743034 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.742941    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c2aa8c-f081-47a8-aec2-eb84c7c9dda7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr\" (UID: \"38c2aa8c-f081-47a8-aec2-eb84c7c9dda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr"
Sep 15 23:01:45.743666 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.743646    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c2aa8c-f081-47a8-aec2-eb84c7c9dda7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr\" (UID: \"38c2aa8c-f081-47a8-aec2-eb84c7c9dda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr"
Sep 15 23:01:45.766146 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.766119    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:45.805988 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:45.805962    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr"
Sep 15 23:01:46.083709 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:46.083684    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr"]
Sep 15 23:01:46.086184 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:01:46.086159    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c2aa8c_f081_47a8_aec2_eb84c7c9dda7.slice/crio-373a13a64cdb539c93b49b03cfedc29ad357e4e0a2120265cda03553d104d76f WatchSource:0}: Error finding container 373a13a64cdb539c93b49b03cfedc29ad357e4e0a2120265cda03553d104d76f: Status 404 returned error can't find the container with id 373a13a64cdb539c93b49b03cfedc29ad357e4e0a2120265cda03553d104d76f
Sep 15 23:01:46.363316 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:46.363231    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr" event={"ID":"38c2aa8c-f081-47a8-aec2-eb84c7c9dda7","Type":"ContainerStarted","Data":"70c91acb3e036717e9e38beeb18d20a0dff0f1b3f034808743e0ac44434b3834"}
Sep 15 23:01:46.363446 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:46.363318    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr" event={"ID":"38c2aa8c-f081-47a8-aec2-eb84c7c9dda7","Type":"ContainerStarted","Data":"373a13a64cdb539c93b49b03cfedc29ad357e4e0a2120265cda03553d104d76f"}
Sep 15 23:01:46.381010 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:46.380958    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr" podStartSLOduration=1.380940561 podStartE2EDuration="1.380940561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 23:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 23:01:46.378684628 +0000 UTC m=+2078.997499547" watchObservedRunningTime="2026-09-15 23:01:46.380940561 +0000 UTC m=+2078.999755470"
Sep 15 23:01:46.673821 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:46.673736    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:47.334805 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:47.334767    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:47.370842 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:47.370811    2584 generic.go:356] "Generic (PLEG): container finished" podID="38c2aa8c-f081-47a8-aec2-eb84c7c9dda7" containerID="70c91acb3e036717e9e38beeb18d20a0dff0f1b3f034808743e0ac44434b3834" exitCode=0
Sep 15 23:01:47.371023 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:47.370901    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr" event={"ID":"38c2aa8c-f081-47a8-aec2-eb84c7c9dda7","Type":"ContainerDied","Data":"70c91acb3e036717e9e38beeb18d20a0dff0f1b3f034808743e0ac44434b3834"}
Sep 15 23:01:48.137608 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:48.137566    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:48.517464 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:48.517437    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr"
Sep 15 23:01:48.666456 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:48.666423    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c2aa8c-f081-47a8-aec2-eb84c7c9dda7-maas-api-service-ca\") pod \"38c2aa8c-f081-47a8-aec2-eb84c7c9dda7\" (UID: \"38c2aa8c-f081-47a8-aec2-eb84c7c9dda7\") "
Sep 15 23:01:48.666795 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:48.666768    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c2aa8c-f081-47a8-aec2-eb84c7c9dda7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38c2aa8c-f081-47a8-aec2-eb84c7c9dda7" (UID: "38c2aa8c-f081-47a8-aec2-eb84c7c9dda7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 23:01:48.768011 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:48.767925    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c2aa8c-f081-47a8-aec2-eb84c7c9dda7-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:01:48.996605 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:48.996575    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:49.383856 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:49.383818    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr"
Sep 15 23:01:49.383856 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:49.383859    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr" event={"ID":"38c2aa8c-f081-47a8-aec2-eb84c7c9dda7","Type":"ContainerDied","Data":"373a13a64cdb539c93b49b03cfedc29ad357e4e0a2120265cda03553d104d76f"}
Sep 15 23:01:49.384087 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:49.383907    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="373a13a64cdb539c93b49b03cfedc29ad357e4e0a2120265cda03553d104d76f"
Sep 15 23:01:50.331714 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.331653    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 23:01:50.390993 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.390954    2584 generic.go:356] "Generic (PLEG): container finished" podID="51454f48-4753-4ebf-9979-26f01e42e967" containerID="31636ee71f25902d9ee50cef7188ae8fbcbd386e7b707c0af2cf178c12c7cc47" exitCode=0
Sep 15 23:01:50.391113 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.391057    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"
Sep 15 23:01:50.391113 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.391088    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" event={"ID":"51454f48-4753-4ebf-9979-26f01e42e967","Type":"ContainerDied","Data":"31636ee71f25902d9ee50cef7188ae8fbcbd386e7b707c0af2cf178c12c7cc47"}
Sep 15 23:01:50.391198 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.391144    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5" event={"ID":"51454f48-4753-4ebf-9979-26f01e42e967","Type":"ContainerDied","Data":"97681c56dd2dac6a230a460b16b518de36559ca035759be458d7d8d56190af2f"}
Sep 15 23:01:50.391198 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.391167    2584 scope.go:117] "RemoveContainer" containerID="31636ee71f25902d9ee50cef7188ae8fbcbd386e7b707c0af2cf178c12c7cc47"
Sep 15 23:01:50.400157 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.400142    2584 scope.go:117] "RemoveContainer" containerID="5bc3bc66b141298672362cac159445e7895c80bad5b44b70a3aaef289d161e8c"
Sep 15 23:01:50.408016 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.407971    2584 scope.go:117] "RemoveContainer" containerID="31636ee71f25902d9ee50cef7188ae8fbcbd386e7b707c0af2cf178c12c7cc47"
Sep 15 23:01:50.408233 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:01:50.408213    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31636ee71f25902d9ee50cef7188ae8fbcbd386e7b707c0af2cf178c12c7cc47\": container with ID starting with 31636ee71f25902d9ee50cef7188ae8fbcbd386e7b707c0af2cf178c12c7cc47 not found: ID does not exist" containerID="31636ee71f25902d9ee50cef7188ae8fbcbd386e7b707c0af2cf178c12c7cc47"
Sep 15 23:01:50.408303 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.408242    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31636ee71f25902d9ee50cef7188ae8fbcbd386e7b707c0af2cf178c12c7cc47"} err="failed to get container status \"31636ee71f25902d9ee50cef7188ae8fbcbd386e7b707c0af2cf178c12c7cc47\": rpc error: code = NotFound desc = could not find container \"31636ee71f25902d9ee50cef7188ae8fbcbd386e7b707c0af2cf178c12c7cc47\": container with ID starting with 31636ee71f25902d9ee50cef7188ae8fbcbd386e7b707c0af2cf178c12c7cc47 not found: ID does not exist"
Sep 15 23:01:50.408303 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.408258    2584 scope.go:117] "RemoveContainer" containerID="5bc3bc66b141298672362cac159445e7895c80bad5b44b70a3aaef289d161e8c"
Sep 15 23:01:50.408536 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:01:50.408518    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc3bc66b141298672362cac159445e7895c80bad5b44b70a3aaef289d161e8c\": container with ID starting with 5bc3bc66b141298672362cac159445e7895c80bad5b44b70a3aaef289d161e8c not found: ID does not exist" containerID="5bc3bc66b141298672362cac159445e7895c80bad5b44b70a3aaef289d161e8c"
Sep 15 23:01:50.408586 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.408550    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc3bc66b141298672362cac159445e7895c80bad5b44b70a3aaef289d161e8c"} err="failed to get container status \"5bc3bc66b141298672362cac159445e7895c80bad5b44b70a3aaef289d161e8c\": rpc error: code = NotFound desc = could not find container \"5bc3bc66b141298672362cac159445e7895c80bad5b44b70a3aaef289d161e8c\": container with ID starting with 5bc3bc66b141298672362cac159445e7895c80bad5b44b70a3aaef289d161e8c not found: ID does not exist"
Sep 15 23:01:50.483541 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.483519    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-istio-envoy\") pod \"51454f48-4753-4ebf-9979-26f01e42e967\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") "
Sep 15 23:01:50.483661 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.483557    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljglt\" (UniqueName: \"kubernetes.io/projected/51454f48-4753-4ebf-9979-26f01e42e967-kube-api-access-ljglt\") pod \"51454f48-4753-4ebf-9979-26f01e42e967\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") "
Sep 15 23:01:50.483661 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.483579    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-workload-certs\") pod \"51454f48-4753-4ebf-9979-26f01e42e967\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") "
Sep 15 23:01:50.483661 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.483599    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-credential-socket\") pod \"51454f48-4753-4ebf-9979-26f01e42e967\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") "
Sep 15 23:01:50.483661 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.483627    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51454f48-4753-4ebf-9979-26f01e42e967-istiod-ca-cert\") pod \"51454f48-4753-4ebf-9979-26f01e42e967\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") "
Sep 15 23:01:50.483876 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.483688    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51454f48-4753-4ebf-9979-26f01e42e967-istio-token\") pod \"51454f48-4753-4ebf-9979-26f01e42e967\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") "
Sep 15 23:01:50.483876 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.483719    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51454f48-4753-4ebf-9979-26f01e42e967-istio-podinfo\") pod \"51454f48-4753-4ebf-9979-26f01e42e967\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") "
Sep 15 23:01:50.483876 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.483775    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-istio-data\") pod \"51454f48-4753-4ebf-9979-26f01e42e967\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") "
Sep 15 23:01:50.483876 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.483804    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-workload-socket\") pod \"51454f48-4753-4ebf-9979-26f01e42e967\" (UID: \"51454f48-4753-4ebf-9979-26f01e42e967\") "
Sep 15 23:01:50.484090 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.483894    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51454f48-4753-4ebf-9979-26f01e42e967" (UID: "51454f48-4753-4ebf-9979-26f01e42e967"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:01:50.484090 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.483913    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51454f48-4753-4ebf-9979-26f01e42e967" (UID: "51454f48-4753-4ebf-9979-26f01e42e967"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:01:50.484194 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.484148    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-istio-data" (OuterVolumeSpecName: "istio-data") pod "51454f48-4753-4ebf-9979-26f01e42e967" (UID: "51454f48-4753-4ebf-9979-26f01e42e967"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:01:50.484194 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.484161    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51454f48-4753-4ebf-9979-26f01e42e967" (UID: "51454f48-4753-4ebf-9979-26f01e42e967"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:01:50.484347 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.484184    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51454f48-4753-4ebf-9979-26f01e42e967-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51454f48-4753-4ebf-9979-26f01e42e967" (UID: "51454f48-4753-4ebf-9979-26f01e42e967"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 23:01:50.484347 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.484330    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-workload-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:01:50.484451 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.484352    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-credential-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:01:50.484451 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.484368    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51454f48-4753-4ebf-9979-26f01e42e967-istiod-ca-cert\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:01:50.484451 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.484382    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-istio-data\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:01:50.484451 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.484395    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-workload-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:01:50.485789 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.485760    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51454f48-4753-4ebf-9979-26f01e42e967" (UID: "51454f48-4753-4ebf-9979-26f01e42e967"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:01:50.485886 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.485810    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51454f48-4753-4ebf-9979-26f01e42e967-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51454f48-4753-4ebf-9979-26f01e42e967" (UID: "51454f48-4753-4ebf-9979-26f01e42e967"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 23:01:50.486153 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.486130    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51454f48-4753-4ebf-9979-26f01e42e967-kube-api-access-ljglt" (OuterVolumeSpecName: "kube-api-access-ljglt") pod "51454f48-4753-4ebf-9979-26f01e42e967" (UID: "51454f48-4753-4ebf-9979-26f01e42e967"). InnerVolumeSpecName "kube-api-access-ljglt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 23:01:50.486233 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.486177    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51454f48-4753-4ebf-9979-26f01e42e967-istio-token" (OuterVolumeSpecName: "istio-token") pod "51454f48-4753-4ebf-9979-26f01e42e967" (UID: "51454f48-4753-4ebf-9979-26f01e42e967"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 23:01:50.585524 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.585447    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51454f48-4753-4ebf-9979-26f01e42e967-istio-token\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:01:50.585524 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.585473    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51454f48-4753-4ebf-9979-26f01e42e967-istio-podinfo\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:01:50.585524 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.585487    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51454f48-4753-4ebf-9979-26f01e42e967-istio-envoy\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:01:50.585524 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.585499    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljglt\" (UniqueName: \"kubernetes.io/projected/51454f48-4753-4ebf-9979-26f01e42e967-kube-api-access-ljglt\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:01:50.714493 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.714450    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"]
Sep 15 23:01:50.718873 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:50.718843    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-637a6f-openshift-default-7d6fbcc68d-xmzx5"]
Sep 15 23:01:51.899495 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:51.899464    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51454f48-4753-4ebf-9979-26f01e42e967" path="/var/lib/kubelet/pods/51454f48-4753-4ebf-9979-26f01e42e967/volumes"
Sep 15 23:01:56.088652 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:56.088600    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-668e8d-29825220-hr2xr"]
Sep 15 23:01:56.090826 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:56.090795    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-668e8d-29825220-hr2xr"]
Sep 15 23:01:56.093169 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:56.093144    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:56.158921 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:56.158893    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:56.221391 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:56.221355    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:56.285756 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:56.285716    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr"]
Sep 15 23:01:56.287689 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:56.287661    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-668e8d-sfhdr"]
Sep 15 23:01:56.291667 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:56.291643    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:56.352457 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:56.352362    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:56.675169 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:56.675074    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:56.758901 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:56.758865    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:57.465528 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:57.465495    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:57.898906 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:57.898877    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c775994-0209-4c09-b63b-dc84b0331359" path="/var/lib/kubelet/pods/1c775994-0209-4c09-b63b-dc84b0331359/volumes"
Sep 15 23:01:57.899206 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:57.899187    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38c2aa8c-f081-47a8-aec2-eb84c7c9dda7" path="/var/lib/kubelet/pods/38c2aa8c-f081-47a8-aec2-eb84c7c9dda7/volumes"
Sep 15 23:01:58.167791 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:58.167695    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:01:59.364078 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:01:59.364042    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:02:02.339246 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.339209    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:02.521576 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.521539    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:02.655346 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.655232    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"]
Sep 15 23:02:02.655533 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.655502    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" podUID="32ac9170-704f-4f09-be27-f875af3abe20" containerName="istio-proxy" containerID="cri-o://25c60cbfc6dabc53cf7a51ec039ba7c86be818f238af5e9b00af868a5ea4bf20" gracePeriod=30
Sep 15 23:02:02.701671 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.701631    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:02.964912 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.964826    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l"]
Sep 15 23:02:02.965208 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.965193    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51454f48-4753-4ebf-9979-26f01e42e967" containerName="istio-proxy"
Sep 15 23:02:02.965208 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.965207    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="51454f48-4753-4ebf-9979-26f01e42e967" containerName="istio-proxy"
Sep 15 23:02:02.965347 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.965221    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38c2aa8c-f081-47a8-aec2-eb84c7c9dda7" containerName="revoke-keys"
Sep 15 23:02:02.965347 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.965227    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="38c2aa8c-f081-47a8-aec2-eb84c7c9dda7" containerName="revoke-keys"
Sep 15 23:02:02.965347 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.965235    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51454f48-4753-4ebf-9979-26f01e42e967" containerName="istio-proxy"
Sep 15 23:02:02.965347 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.965241    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="51454f48-4753-4ebf-9979-26f01e42e967" containerName="istio-proxy"
Sep 15 23:02:02.965347 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.965329    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="51454f48-4753-4ebf-9979-26f01e42e967" containerName="istio-proxy"
Sep 15 23:02:02.965347 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.965339    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="38c2aa8c-f081-47a8-aec2-eb84c7c9dda7" containerName="revoke-keys"
Sep 15 23:02:02.969730 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.969713    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l"
Sep 15 23:02:02.972468 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.972444    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 23:02:02.972591 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.972569    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 23:02:02.979169 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.979105    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l"]
Sep 15 23:02:02.994587 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:02.994550    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b7145c-80b6-4647-8022-8c53469cc785-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l\" (UID: \"33b7145c-80b6-4647-8022-8c53469cc785\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l"
Sep 15 23:02:03.090191 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:03.090160    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:03.095879 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:03.095851    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b7145c-80b6-4647-8022-8c53469cc785-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l\" (UID: \"33b7145c-80b6-4647-8022-8c53469cc785\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l"
Sep 15 23:02:03.096694 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:03.096669    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b7145c-80b6-4647-8022-8c53469cc785-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l\" (UID: \"33b7145c-80b6-4647-8022-8c53469cc785\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l"
Sep 15 23:02:03.284591 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:03.284479    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l"
Sep 15 23:02:03.566955 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:03.566925    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l"]
Sep 15 23:02:03.569387 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:02:03.569348    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b7145c_80b6_4647_8022_8c53469cc785.slice/crio-bc85b86ba3ee242e2b31b44fafdd6643d7f99b1d9030d0e5b8714c9c3335c5d0 WatchSource:0}: Error finding container bc85b86ba3ee242e2b31b44fafdd6643d7f99b1d9030d0e5b8714c9c3335c5d0: Status 404 returned error can't find the container with id bc85b86ba3ee242e2b31b44fafdd6643d7f99b1d9030d0e5b8714c9c3335c5d0
Sep 15 23:02:04.042027 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:04.041998    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:04.472463 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:04.472422    2584 generic.go:356] "Generic (PLEG): container finished" podID="33b7145c-80b6-4647-8022-8c53469cc785" containerID="696b8f833ca6089946c206ac0cf6f0a3d73aad3135a640991b361302f89ff595" exitCode=0
Sep 15 23:02:04.472648 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:04.472477    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l" event={"ID":"33b7145c-80b6-4647-8022-8c53469cc785","Type":"ContainerDied","Data":"696b8f833ca6089946c206ac0cf6f0a3d73aad3135a640991b361302f89ff595"}
Sep 15 23:02:04.472648 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:04.472527    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l" event={"ID":"33b7145c-80b6-4647-8022-8c53469cc785","Type":"ContainerStarted","Data":"bc85b86ba3ee242e2b31b44fafdd6643d7f99b1d9030d0e5b8714c9c3335c5d0"}
Sep 15 23:02:05.629584 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:05.629562    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l"
Sep 15 23:02:05.720175 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:05.720128    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b7145c-80b6-4647-8022-8c53469cc785-maas-api-service-ca\") pod \"33b7145c-80b6-4647-8022-8c53469cc785\" (UID: \"33b7145c-80b6-4647-8022-8c53469cc785\") "
Sep 15 23:02:05.720493 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:05.720467    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b7145c-80b6-4647-8022-8c53469cc785-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33b7145c-80b6-4647-8022-8c53469cc785" (UID: "33b7145c-80b6-4647-8022-8c53469cc785"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 23:02:05.720600 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:05.720587    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b7145c-80b6-4647-8022-8c53469cc785-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:06.023789 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:06.023751    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:06.484159 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:06.484115    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l" event={"ID":"33b7145c-80b6-4647-8022-8c53469cc785","Type":"ContainerDied","Data":"bc85b86ba3ee242e2b31b44fafdd6643d7f99b1d9030d0e5b8714c9c3335c5d0"}
Sep 15 23:02:06.484326 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:06.484171    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc85b86ba3ee242e2b31b44fafdd6643d7f99b1d9030d0e5b8714c9c3335c5d0"
Sep 15 23:02:06.484326 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:06.484137    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l"
Sep 15 23:02:07.906191 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.906169    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 23:02:07.939006 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.938938    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgpwd\" (UniqueName: \"kubernetes.io/projected/32ac9170-704f-4f09-be27-f875af3abe20-kube-api-access-pgpwd\") pod \"32ac9170-704f-4f09-be27-f875af3abe20\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") "
Sep 15 23:02:07.939130 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939013    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ac9170-704f-4f09-be27-f875af3abe20-istio-token\") pod \"32ac9170-704f-4f09-be27-f875af3abe20\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") "
Sep 15 23:02:07.939130 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939095    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-workload-certs\") pod \"32ac9170-704f-4f09-be27-f875af3abe20\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") "
Sep 15 23:02:07.939232 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939130    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-workload-socket\") pod \"32ac9170-704f-4f09-be27-f875af3abe20\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") "
Sep 15 23:02:07.939232 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939163    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-istio-data\") pod \"32ac9170-704f-4f09-be27-f875af3abe20\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") "
Sep 15 23:02:07.939232 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939188    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-istio-envoy\") pod \"32ac9170-704f-4f09-be27-f875af3abe20\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") "
Sep 15 23:02:07.939408 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939243    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-credential-socket\") pod \"32ac9170-704f-4f09-be27-f875af3abe20\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") "
Sep 15 23:02:07.939408 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939318    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ac9170-704f-4f09-be27-f875af3abe20-istio-podinfo\") pod \"32ac9170-704f-4f09-be27-f875af3abe20\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") "
Sep 15 23:02:07.939408 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939346    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ac9170-704f-4f09-be27-f875af3abe20-istiod-ca-cert\") pod \"32ac9170-704f-4f09-be27-f875af3abe20\" (UID: \"32ac9170-704f-4f09-be27-f875af3abe20\") "
Sep 15 23:02:07.939556 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939447    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "32ac9170-704f-4f09-be27-f875af3abe20" (UID: "32ac9170-704f-4f09-be27-f875af3abe20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:02:07.939556 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939456    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "32ac9170-704f-4f09-be27-f875af3abe20" (UID: "32ac9170-704f-4f09-be27-f875af3abe20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:02:07.939556 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939466    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-istio-data" (OuterVolumeSpecName: "istio-data") pod "32ac9170-704f-4f09-be27-f875af3abe20" (UID: "32ac9170-704f-4f09-be27-f875af3abe20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:02:07.939695 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939669    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "32ac9170-704f-4f09-be27-f875af3abe20" (UID: "32ac9170-704f-4f09-be27-f875af3abe20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:02:07.939802 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939770    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ac9170-704f-4f09-be27-f875af3abe20-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "32ac9170-704f-4f09-be27-f875af3abe20" (UID: "32ac9170-704f-4f09-be27-f875af3abe20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 23:02:07.939885 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939775    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-workload-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:07.939885 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939838    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-workload-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:07.939885 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939856    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-istio-data\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:07.939885 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.939872    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-credential-socket\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:07.941729 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.941703    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "32ac9170-704f-4f09-be27-f875af3abe20" (UID: "32ac9170-704f-4f09-be27-f875af3abe20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:02:07.941729 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.941724    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ac9170-704f-4f09-be27-f875af3abe20-istio-token" (OuterVolumeSpecName: "istio-token") pod "32ac9170-704f-4f09-be27-f875af3abe20" (UID: "32ac9170-704f-4f09-be27-f875af3abe20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 23:02:07.942011 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.941989    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32ac9170-704f-4f09-be27-f875af3abe20-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "32ac9170-704f-4f09-be27-f875af3abe20" (UID: "32ac9170-704f-4f09-be27-f875af3abe20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 23:02:07.942099 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:07.941989    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ac9170-704f-4f09-be27-f875af3abe20-kube-api-access-pgpwd" (OuterVolumeSpecName: "kube-api-access-pgpwd") pod "32ac9170-704f-4f09-be27-f875af3abe20" (UID: "32ac9170-704f-4f09-be27-f875af3abe20"). InnerVolumeSpecName "kube-api-access-pgpwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 23:02:08.021026 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.020997    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 23:02:08.024523 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.024498    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 23:02:08.041295 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.041255    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ac9170-704f-4f09-be27-f875af3abe20-istio-podinfo\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:08.041295 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.041295    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ac9170-704f-4f09-be27-f875af3abe20-istiod-ca-cert\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:08.041436 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.041306    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgpwd\" (UniqueName: \"kubernetes.io/projected/32ac9170-704f-4f09-be27-f875af3abe20-kube-api-access-pgpwd\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:08.041436 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.041315    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ac9170-704f-4f09-be27-f875af3abe20-istio-token\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:08.041436 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.041324    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ac9170-704f-4f09-be27-f875af3abe20-istio-envoy\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:08.495826 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.495795    2584 generic.go:356] "Generic (PLEG): container finished" podID="32ac9170-704f-4f09-be27-f875af3abe20" containerID="25c60cbfc6dabc53cf7a51ec039ba7c86be818f238af5e9b00af868a5ea4bf20" exitCode=0
Sep 15 23:02:08.496012 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.495852    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"
Sep 15 23:02:08.496012 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.495880    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" event={"ID":"32ac9170-704f-4f09-be27-f875af3abe20","Type":"ContainerDied","Data":"25c60cbfc6dabc53cf7a51ec039ba7c86be818f238af5e9b00af868a5ea4bf20"}
Sep 15 23:02:08.496012 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.495947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb" event={"ID":"32ac9170-704f-4f09-be27-f875af3abe20","Type":"ContainerDied","Data":"d7df0019a19bf33b1a5f2bf3ffe326fc2c1d0dbc26388e06d67512fbbe418ed9"}
Sep 15 23:02:08.496012 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.495976    2584 scope.go:117] "RemoveContainer" containerID="25c60cbfc6dabc53cf7a51ec039ba7c86be818f238af5e9b00af868a5ea4bf20"
Sep 15 23:02:08.505045 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.505025    2584 scope.go:117] "RemoveContainer" containerID="7c45018205097481f6d6904aac0b142cb5f5033329e1936b3d1e42f3b36609a4"
Sep 15 23:02:08.512928 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.512909    2584 scope.go:117] "RemoveContainer" containerID="25c60cbfc6dabc53cf7a51ec039ba7c86be818f238af5e9b00af868a5ea4bf20"
Sep 15 23:02:08.513202 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:02:08.513184    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c60cbfc6dabc53cf7a51ec039ba7c86be818f238af5e9b00af868a5ea4bf20\": container with ID starting with 25c60cbfc6dabc53cf7a51ec039ba7c86be818f238af5e9b00af868a5ea4bf20 not found: ID does not exist" containerID="25c60cbfc6dabc53cf7a51ec039ba7c86be818f238af5e9b00af868a5ea4bf20"
Sep 15 23:02:08.513248 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.513210    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c60cbfc6dabc53cf7a51ec039ba7c86be818f238af5e9b00af868a5ea4bf20"} err="failed to get container status \"25c60cbfc6dabc53cf7a51ec039ba7c86be818f238af5e9b00af868a5ea4bf20\": rpc error: code = NotFound desc = could not find container \"25c60cbfc6dabc53cf7a51ec039ba7c86be818f238af5e9b00af868a5ea4bf20\": container with ID starting with 25c60cbfc6dabc53cf7a51ec039ba7c86be818f238af5e9b00af868a5ea4bf20 not found: ID does not exist"
Sep 15 23:02:08.513248 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.513228    2584 scope.go:117] "RemoveContainer" containerID="7c45018205097481f6d6904aac0b142cb5f5033329e1936b3d1e42f3b36609a4"
Sep 15 23:02:08.513476 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:02:08.513459    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c45018205097481f6d6904aac0b142cb5f5033329e1936b3d1e42f3b36609a4\": container with ID starting with 7c45018205097481f6d6904aac0b142cb5f5033329e1936b3d1e42f3b36609a4 not found: ID does not exist" containerID="7c45018205097481f6d6904aac0b142cb5f5033329e1936b3d1e42f3b36609a4"
Sep 15 23:02:08.513518 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.513480    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c45018205097481f6d6904aac0b142cb5f5033329e1936b3d1e42f3b36609a4"} err="failed to get container status \"7c45018205097481f6d6904aac0b142cb5f5033329e1936b3d1e42f3b36609a4\": rpc error: code = NotFound desc = could not find container \"7c45018205097481f6d6904aac0b142cb5f5033329e1936b3d1e42f3b36609a4\": container with ID starting with 7c45018205097481f6d6904aac0b142cb5f5033329e1936b3d1e42f3b36609a4 not found: ID does not exist"
Sep 15 23:02:08.519341 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.519319    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"]
Sep 15 23:02:08.522975 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:08.522955    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-668e8d-openshift-default-7466677b94-q9qpb"]
Sep 15 23:02:09.901857 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:09.901822    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ac9170-704f-4f09-be27-f875af3abe20" path="/var/lib/kubelet/pods/32ac9170-704f-4f09-be27-f875af3abe20/volumes"
Sep 15 23:02:13.507088 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.507045    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:13.540198 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.540150    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"]
Sep 15 23:02:13.540446 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.540413    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf" podUID="82d084a5-4f09-4188-8d16-57db849fb124" containerName="payload-processing" containerID="cri-o://75972eb6c310ad7483078c50a3809f75d37f30bb63aa2107cfe9b9aff8a61e23" gracePeriod=30
Sep 15 23:02:13.549666 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.549639    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"]
Sep 15 23:02:13.550065 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.550023    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk" podUID="f7c420a4-2764-4622-b52f-e48999b2fbb6" containerName="payload-pre-processing" containerID="cri-o://03508bfd029a3b8f70b09ec20977c3bbf496728fff9f8e04f6bf733209c49fd3" gracePeriod=30
Sep 15 23:02:13.585235 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.585201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:13.645422 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.645397    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:13.717326 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.717294    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l"]
Sep 15 23:02:13.718525 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.718494    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:13.720243 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.720216    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0016d4-j4t6l"]
Sep 15 23:02:13.790166 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.790088    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:13.828804 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.828782    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"
Sep 15 23:02:13.831614 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.831594    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"
Sep 15 23:02:13.887423 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.887392    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rxm6\" (UniqueName: \"kubernetes.io/projected/f7c420a4-2764-4622-b52f-e48999b2fbb6-kube-api-access-5rxm6\") pod \"f7c420a4-2764-4622-b52f-e48999b2fbb6\" (UID: \"f7c420a4-2764-4622-b52f-e48999b2fbb6\") "
Sep 15 23:02:13.887669 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.887442    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wd77\" (UniqueName: \"kubernetes.io/projected/82d084a5-4f09-4188-8d16-57db849fb124-kube-api-access-6wd77\") pod \"82d084a5-4f09-4188-8d16-57db849fb124\" (UID: \"82d084a5-4f09-4188-8d16-57db849fb124\") "
Sep 15 23:02:13.887669 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.887483    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c420a4-2764-4622-b52f-e48999b2fbb6-plugins-config-volume\") pod \"f7c420a4-2764-4622-b52f-e48999b2fbb6\" (UID: \"f7c420a4-2764-4622-b52f-e48999b2fbb6\") "
Sep 15 23:02:13.887669 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.887596    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d084a5-4f09-4188-8d16-57db849fb124-plugins-config-volume\") pod \"82d084a5-4f09-4188-8d16-57db849fb124\" (UID: \"82d084a5-4f09-4188-8d16-57db849fb124\") "
Sep 15 23:02:13.888292 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.888204    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d084a5-4f09-4188-8d16-57db849fb124-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82d084a5-4f09-4188-8d16-57db849fb124" (UID: "82d084a5-4f09-4188-8d16-57db849fb124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 23:02:13.888552 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.888529    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c420a4-2764-4622-b52f-e48999b2fbb6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7c420a4-2764-4622-b52f-e48999b2fbb6" (UID: "f7c420a4-2764-4622-b52f-e48999b2fbb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 23:02:13.889643 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.889622    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c420a4-2764-4622-b52f-e48999b2fbb6-kube-api-access-5rxm6" (OuterVolumeSpecName: "kube-api-access-5rxm6") pod "f7c420a4-2764-4622-b52f-e48999b2fbb6" (UID: "f7c420a4-2764-4622-b52f-e48999b2fbb6"). InnerVolumeSpecName "kube-api-access-5rxm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 23:02:13.889821 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.889800    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d084a5-4f09-4188-8d16-57db849fb124-kube-api-access-6wd77" (OuterVolumeSpecName: "kube-api-access-6wd77") pod "82d084a5-4f09-4188-8d16-57db849fb124" (UID: "82d084a5-4f09-4188-8d16-57db849fb124"). InnerVolumeSpecName "kube-api-access-6wd77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 23:02:13.901064 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.901034    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b7145c-80b6-4647-8022-8c53469cc785" path="/var/lib/kubelet/pods/33b7145c-80b6-4647-8022-8c53469cc785/volumes"
Sep 15 23:02:13.988682 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.988648    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d084a5-4f09-4188-8d16-57db849fb124-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:13.988682 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.988676    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rxm6\" (UniqueName: \"kubernetes.io/projected/f7c420a4-2764-4622-b52f-e48999b2fbb6-kube-api-access-5rxm6\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:13.988915 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.988690    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wd77\" (UniqueName: \"kubernetes.io/projected/82d084a5-4f09-4188-8d16-57db849fb124-kube-api-access-6wd77\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:13.988915 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:13.988704    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c420a4-2764-4622-b52f-e48999b2fbb6-plugins-config-volume\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:14.087519 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.087477    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:14.187679 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.187651    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:14.528906 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.528816    2584 generic.go:356] "Generic (PLEG): container finished" podID="f7c420a4-2764-4622-b52f-e48999b2fbb6" containerID="03508bfd029a3b8f70b09ec20977c3bbf496728fff9f8e04f6bf733209c49fd3" exitCode=0
Sep 15 23:02:14.528906 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.528870    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"
Sep 15 23:02:14.529444 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.528903    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk" event={"ID":"f7c420a4-2764-4622-b52f-e48999b2fbb6","Type":"ContainerDied","Data":"03508bfd029a3b8f70b09ec20977c3bbf496728fff9f8e04f6bf733209c49fd3"}
Sep 15 23:02:14.529444 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.528974    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk" event={"ID":"f7c420a4-2764-4622-b52f-e48999b2fbb6","Type":"ContainerDied","Data":"ef59b0194d0ea3937f63fb478b02ce20aa4e3b375f866bfbbd070bce5d55e9f8"}
Sep 15 23:02:14.529444 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.529011    2584 scope.go:117] "RemoveContainer" containerID="03508bfd029a3b8f70b09ec20977c3bbf496728fff9f8e04f6bf733209c49fd3"
Sep 15 23:02:14.530430 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.530407    2584 generic.go:356] "Generic (PLEG): container finished" podID="82d084a5-4f09-4188-8d16-57db849fb124" containerID="75972eb6c310ad7483078c50a3809f75d37f30bb63aa2107cfe9b9aff8a61e23" exitCode=0
Sep 15 23:02:14.530546 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.530455    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"
Sep 15 23:02:14.530546 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.530521    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf" event={"ID":"82d084a5-4f09-4188-8d16-57db849fb124","Type":"ContainerDied","Data":"75972eb6c310ad7483078c50a3809f75d37f30bb63aa2107cfe9b9aff8a61e23"}
Sep 15 23:02:14.530665 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.530561    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf" event={"ID":"82d084a5-4f09-4188-8d16-57db849fb124","Type":"ContainerDied","Data":"6abb77987d0b259f64cf5f3e150f99651f7a38c6cf1f8afee67b1ab34f72a2d4"}
Sep 15 23:02:14.538133 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.538115    2584 scope.go:117] "RemoveContainer" containerID="03508bfd029a3b8f70b09ec20977c3bbf496728fff9f8e04f6bf733209c49fd3"
Sep 15 23:02:14.538449 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:02:14.538430    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03508bfd029a3b8f70b09ec20977c3bbf496728fff9f8e04f6bf733209c49fd3\": container with ID starting with 03508bfd029a3b8f70b09ec20977c3bbf496728fff9f8e04f6bf733209c49fd3 not found: ID does not exist" containerID="03508bfd029a3b8f70b09ec20977c3bbf496728fff9f8e04f6bf733209c49fd3"
Sep 15 23:02:14.538501 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.538457    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03508bfd029a3b8f70b09ec20977c3bbf496728fff9f8e04f6bf733209c49fd3"} err="failed to get container status \"03508bfd029a3b8f70b09ec20977c3bbf496728fff9f8e04f6bf733209c49fd3\": rpc error: code = NotFound desc = could not find container \"03508bfd029a3b8f70b09ec20977c3bbf496728fff9f8e04f6bf733209c49fd3\": container with ID starting with 03508bfd029a3b8f70b09ec20977c3bbf496728fff9f8e04f6bf733209c49fd3 not found: ID does not exist"
Sep 15 23:02:14.538501 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.538474    2584 scope.go:117] "RemoveContainer" containerID="75972eb6c310ad7483078c50a3809f75d37f30bb63aa2107cfe9b9aff8a61e23"
Sep 15 23:02:14.547311 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.547125    2584 scope.go:117] "RemoveContainer" containerID="75972eb6c310ad7483078c50a3809f75d37f30bb63aa2107cfe9b9aff8a61e23"
Sep 15 23:02:14.547432 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:02:14.547410    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75972eb6c310ad7483078c50a3809f75d37f30bb63aa2107cfe9b9aff8a61e23\": container with ID starting with 75972eb6c310ad7483078c50a3809f75d37f30bb63aa2107cfe9b9aff8a61e23 not found: ID does not exist" containerID="75972eb6c310ad7483078c50a3809f75d37f30bb63aa2107cfe9b9aff8a61e23"
Sep 15 23:02:14.547477 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.547443    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75972eb6c310ad7483078c50a3809f75d37f30bb63aa2107cfe9b9aff8a61e23"} err="failed to get container status \"75972eb6c310ad7483078c50a3809f75d37f30bb63aa2107cfe9b9aff8a61e23\": rpc error: code = NotFound desc = could not find container \"75972eb6c310ad7483078c50a3809f75d37f30bb63aa2107cfe9b9aff8a61e23\": container with ID starting with 75972eb6c310ad7483078c50a3809f75d37f30bb63aa2107cfe9b9aff8a61e23 not found: ID does not exist"
Sep 15 23:02:14.550169 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.550144    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"]
Sep 15 23:02:14.556152 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.556132    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0016d4-7c7b9c49bb-9sfhk"]
Sep 15 23:02:14.573288 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.573255    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"]
Sep 15 23:02:14.581243 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.581219    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0016d4-645c4c996f-fzcvf"]
Sep 15 23:02:14.968251 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:14.968210    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:15.898283 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:15.898237    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82d084a5-4f09-4188-8d16-57db849fb124" path="/var/lib/kubelet/pods/82d084a5-4f09-4188-8d16-57db849fb124/volumes"
Sep 15 23:02:15.898646 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:15.898631    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7c420a4-2764-4622-b52f-e48999b2fbb6" path="/var/lib/kubelet/pods/f7c420a4-2764-4622-b52f-e48999b2fbb6/volumes"
Sep 15 23:02:16.364302 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:16.364256    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:02:20.681747 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:20.681716    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:20.783750 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:20.783717    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:20.967892 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:20.967807    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:21.282753 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.282669    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm"]
Sep 15 23:02:21.283086 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283063    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ac9170-704f-4f09-be27-f875af3abe20" containerName="istio-proxy"
Sep 15 23:02:21.283086 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283080    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ac9170-704f-4f09-be27-f875af3abe20" containerName="istio-proxy"
Sep 15 23:02:21.283257 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283095    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b7145c-80b6-4647-8022-8c53469cc785" containerName="revoke-keys"
Sep 15 23:02:21.283257 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283101    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b7145c-80b6-4647-8022-8c53469cc785" containerName="revoke-keys"
Sep 15 23:02:21.283257 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283119    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ac9170-704f-4f09-be27-f875af3abe20" containerName="istio-proxy"
Sep 15 23:02:21.283257 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283125    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ac9170-704f-4f09-be27-f875af3abe20" containerName="istio-proxy"
Sep 15 23:02:21.283257 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283140    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c420a4-2764-4622-b52f-e48999b2fbb6" containerName="payload-pre-processing"
Sep 15 23:02:21.283257 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283145    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c420a4-2764-4622-b52f-e48999b2fbb6" containerName="payload-pre-processing"
Sep 15 23:02:21.283257 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283154    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d084a5-4f09-4188-8d16-57db849fb124" containerName="payload-processing"
Sep 15 23:02:21.283257 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283159    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d084a5-4f09-4188-8d16-57db849fb124" containerName="payload-processing"
Sep 15 23:02:21.283257 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283220    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d084a5-4f09-4188-8d16-57db849fb124" containerName="payload-processing"
Sep 15 23:02:21.283257 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283237    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b7145c-80b6-4647-8022-8c53469cc785" containerName="revoke-keys"
Sep 15 23:02:21.283257 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283242    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c420a4-2764-4622-b52f-e48999b2fbb6" containerName="payload-pre-processing"
Sep 15 23:02:21.283257 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283248    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ac9170-704f-4f09-be27-f875af3abe20" containerName="istio-proxy"
Sep 15 23:02:21.283257 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.283255    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="51454f48-4753-4ebf-9979-26f01e42e967" containerName="istio-proxy"
Sep 15 23:02:21.288130 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.288109    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm"
Sep 15 23:02:21.291291 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.291242    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 23:02:21.291418 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.291252    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnj7x\""
Sep 15 23:02:21.296944 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.296918    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm"]
Sep 15 23:02:21.355441 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.355406    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/841e37f5-e0bd-40c5-9b2e-f357b61f9881-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm\" (UID: \"841e37f5-e0bd-40c5-9b2e-f357b61f9881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm"
Sep 15 23:02:21.456591 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.456555    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/841e37f5-e0bd-40c5-9b2e-f357b61f9881-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm\" (UID: \"841e37f5-e0bd-40c5-9b2e-f357b61f9881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm"
Sep 15 23:02:21.457219 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.457194    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/841e37f5-e0bd-40c5-9b2e-f357b61f9881-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm\" (UID: \"841e37f5-e0bd-40c5-9b2e-f357b61f9881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm"
Sep 15 23:02:21.601451 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.601424    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm"
Sep 15 23:02:21.872615 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:21.872586    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm"]
Sep 15 23:02:21.874529 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:02:21.874500    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841e37f5_e0bd_40c5_9b2e_f357b61f9881.slice/crio-8d4f2eebe40adac0e349348f7681ade4b1215b3eec911a7d88fd056217830418 WatchSource:0}: Error finding container 8d4f2eebe40adac0e349348f7681ade4b1215b3eec911a7d88fd056217830418: Status 404 returned error can't find the container with id 8d4f2eebe40adac0e349348f7681ade4b1215b3eec911a7d88fd056217830418
Sep 15 23:02:22.271737 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:22.271703    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:22.576671 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:22.576640    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm" event={"ID":"841e37f5-e0bd-40c5-9b2e-f357b61f9881","Type":"ContainerStarted","Data":"8bb4c48bffde11ad41a59e861f88a1c7f481f8866ab8f11a684e9da509377b09"}
Sep 15 23:02:22.576851 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:22.576694    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm" event={"ID":"841e37f5-e0bd-40c5-9b2e-f357b61f9881","Type":"ContainerStarted","Data":"8d4f2eebe40adac0e349348f7681ade4b1215b3eec911a7d88fd056217830418"}
Sep 15 23:02:22.594178 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:22.594133    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm" podStartSLOduration=1.594119954 podStartE2EDuration="1.594119954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 23:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 23:02:22.592958946 +0000 UTC m=+2115.211773853" watchObservedRunningTime="2026-09-15 23:02:22.594119954 +0000 UTC m=+2115.212934861"
Sep 15 23:02:23.195459 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:23.195427    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:23.584358 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:23.584320    2584 generic.go:356] "Generic (PLEG): container finished" podID="841e37f5-e0bd-40c5-9b2e-f357b61f9881" containerID="8bb4c48bffde11ad41a59e861f88a1c7f481f8866ab8f11a684e9da509377b09" exitCode=0
Sep 15 23:02:23.584533 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:23.584413    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm" event={"ID":"841e37f5-e0bd-40c5-9b2e-f357b61f9881","Type":"ContainerDied","Data":"8bb4c48bffde11ad41a59e861f88a1c7f481f8866ab8f11a684e9da509377b09"}
Sep 15 23:02:24.720872 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:24.720840    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm"
Sep 15 23:02:24.888434 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:24.888352    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/841e37f5-e0bd-40c5-9b2e-f357b61f9881-maas-api-service-ca\") pod \"841e37f5-e0bd-40c5-9b2e-f357b61f9881\" (UID: \"841e37f5-e0bd-40c5-9b2e-f357b61f9881\") "
Sep 15 23:02:24.888744 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:24.888718    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841e37f5-e0bd-40c5-9b2e-f357b61f9881-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "841e37f5-e0bd-40c5-9b2e-f357b61f9881" (UID: "841e37f5-e0bd-40c5-9b2e-f357b61f9881"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 23:02:24.989878 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:24.989846    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/841e37f5-e0bd-40c5-9b2e-f357b61f9881-maas-api-service-ca\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:25.596558 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:25.596532    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm"
Sep 15 23:02:25.596558 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:25.596544    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm" event={"ID":"841e37f5-e0bd-40c5-9b2e-f357b61f9881","Type":"ContainerDied","Data":"8d4f2eebe40adac0e349348f7681ade4b1215b3eec911a7d88fd056217830418"}
Sep 15 23:02:25.596767 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:25.596598    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d4f2eebe40adac0e349348f7681ade4b1215b3eec911a7d88fd056217830418"
Sep 15 23:02:31.955950 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:31.955913    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:32.055130 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:32.055085    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:32.154811 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:32.154778    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:32.162313 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:32.162287    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm"]
Sep 15 23:02:32.166672 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:32.166648    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-92482b-gbdqm"]
Sep 15 23:02:32.255723 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:32.255632    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:32.554979 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:32.554944    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:32.667793 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:32.667763    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:33.899013 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:33.898978    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="841e37f5-e0bd-40c5-9b2e-f357b61f9881" path="/var/lib/kubelet/pods/841e37f5-e0bd-40c5-9b2e-f357b61f9881/volumes"
Sep 15 23:02:34.663232 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:34.663197    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:02:38.420452 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:38.420419    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:38.521692 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:38.521656    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:38.687372 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:38.687284    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:39.113637 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:39.113581    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:02:41.124301 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.124256    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54fd547858-8zt6x"]
Sep 15 23:02:41.124691 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.124461    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-54fd547858-8zt6x" podUID="2485cbbf-64ed-406c-b3a3-c64880fe1be9" containerName="manager" containerID="cri-o://5fd67dd2c285dfdbde827f1d658ee7cb45048e4a618f3ee1699bb6b40d7b1ec6" gracePeriod=30
Sep 15 23:02:41.374614 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.374559    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 23:02:41.432845 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.432814    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2485cbbf-64ed-406c-b3a3-c64880fe1be9-webhook-cert\") pod \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\" (UID: \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\") "
Sep 15 23:02:41.433130 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.433111    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2485cbbf-64ed-406c-b3a3-c64880fe1be9-metrics-certs\") pod \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\" (UID: \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\") "
Sep 15 23:02:41.433373 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.433355    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwx92\" (UniqueName: \"kubernetes.io/projected/2485cbbf-64ed-406c-b3a3-c64880fe1be9-kube-api-access-rwx92\") pod \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\" (UID: \"2485cbbf-64ed-406c-b3a3-c64880fe1be9\") "
Sep 15 23:02:41.435569 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.435233    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2485cbbf-64ed-406c-b3a3-c64880fe1be9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2485cbbf-64ed-406c-b3a3-c64880fe1be9" (UID: "2485cbbf-64ed-406c-b3a3-c64880fe1be9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 23:02:41.436520 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.436490    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2485cbbf-64ed-406c-b3a3-c64880fe1be9-kube-api-access-rwx92" (OuterVolumeSpecName: "kube-api-access-rwx92") pod "2485cbbf-64ed-406c-b3a3-c64880fe1be9" (UID: "2485cbbf-64ed-406c-b3a3-c64880fe1be9"). InnerVolumeSpecName "kube-api-access-rwx92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 23:02:41.440857 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.437774    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2485cbbf-64ed-406c-b3a3-c64880fe1be9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2485cbbf-64ed-406c-b3a3-c64880fe1be9" (UID: "2485cbbf-64ed-406c-b3a3-c64880fe1be9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 23:02:41.534418 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.534393    2584 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2485cbbf-64ed-406c-b3a3-c64880fe1be9-webhook-cert\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:41.534418 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.534415    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2485cbbf-64ed-406c-b3a3-c64880fe1be9-metrics-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:41.534556 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.534424    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwx92\" (UniqueName: \"kubernetes.io/projected/2485cbbf-64ed-406c-b3a3-c64880fe1be9-kube-api-access-rwx92\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:02:41.677583 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.677508    2584 generic.go:356] "Generic (PLEG): container finished" podID="2485cbbf-64ed-406c-b3a3-c64880fe1be9" containerID="5fd67dd2c285dfdbde827f1d658ee7cb45048e4a618f3ee1699bb6b40d7b1ec6" exitCode=0
Sep 15 23:02:41.677583 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.677555    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd547858-8zt6x" event={"ID":"2485cbbf-64ed-406c-b3a3-c64880fe1be9","Type":"ContainerDied","Data":"5fd67dd2c285dfdbde827f1d658ee7cb45048e4a618f3ee1699bb6b40d7b1ec6"}
Sep 15 23:02:41.677583 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.677567    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd547858-8zt6x"
Sep 15 23:02:41.677815 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.677644    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd547858-8zt6x" event={"ID":"2485cbbf-64ed-406c-b3a3-c64880fe1be9","Type":"ContainerDied","Data":"10716b6c2accdee1463cdb43123ca369e208b20f40f4a2420a9b604cc965e1fe"}
Sep 15 23:02:41.677815 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.677677    2584 scope.go:117] "RemoveContainer" containerID="5fd67dd2c285dfdbde827f1d658ee7cb45048e4a618f3ee1699bb6b40d7b1ec6"
Sep 15 23:02:41.687627 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.687610    2584 scope.go:117] "RemoveContainer" containerID="5fd67dd2c285dfdbde827f1d658ee7cb45048e4a618f3ee1699bb6b40d7b1ec6"
Sep 15 23:02:41.687901 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:02:41.687882    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd67dd2c285dfdbde827f1d658ee7cb45048e4a618f3ee1699bb6b40d7b1ec6\": container with ID starting with 5fd67dd2c285dfdbde827f1d658ee7cb45048e4a618f3ee1699bb6b40d7b1ec6 not found: ID does not exist" containerID="5fd67dd2c285dfdbde827f1d658ee7cb45048e4a618f3ee1699bb6b40d7b1ec6"
Sep 15 23:02:41.687967 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.687908    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd67dd2c285dfdbde827f1d658ee7cb45048e4a618f3ee1699bb6b40d7b1ec6"} err="failed to get container status \"5fd67dd2c285dfdbde827f1d658ee7cb45048e4a618f3ee1699bb6b40d7b1ec6\": rpc error: code = NotFound desc = could not find container \"5fd67dd2c285dfdbde827f1d658ee7cb45048e4a618f3ee1699bb6b40d7b1ec6\": container with ID starting with 5fd67dd2c285dfdbde827f1d658ee7cb45048e4a618f3ee1699bb6b40d7b1ec6 not found: ID does not exist"
Sep 15 23:02:41.702017 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.701995    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54fd547858-8zt6x"]
Sep 15 23:02:41.707277 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.707245    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-54fd547858-8zt6x"]
Sep 15 23:02:41.898813 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:41.898781    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2485cbbf-64ed-406c-b3a3-c64880fe1be9" path="/var/lib/kubelet/pods/2485cbbf-64ed-406c-b3a3-c64880fe1be9/volumes"
Sep 15 23:02:43.766676 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:02:43.766641    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:03:20.049339 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:20.049300    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:03:20.108194 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:20.108154    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:03:34.884350 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:34.884303    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:03:42.501562 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.501525    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54fd547858-fdlgn"]
Sep 15 23:03:42.501972 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.501872    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2485cbbf-64ed-406c-b3a3-c64880fe1be9" containerName="manager"
Sep 15 23:03:42.501972 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.501883    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2485cbbf-64ed-406c-b3a3-c64880fe1be9" containerName="manager"
Sep 15 23:03:42.501972 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.501907    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841e37f5-e0bd-40c5-9b2e-f357b61f9881" containerName="revoke-keys"
Sep 15 23:03:42.501972 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.501913    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="841e37f5-e0bd-40c5-9b2e-f357b61f9881" containerName="revoke-keys"
Sep 15 23:03:42.501972 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.501969    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ac9170-704f-4f09-be27-f875af3abe20" containerName="istio-proxy"
Sep 15 23:03:42.502168 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.501977    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="841e37f5-e0bd-40c5-9b2e-f357b61f9881" containerName="revoke-keys"
Sep 15 23:03:42.502168 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.501986    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2485cbbf-64ed-406c-b3a3-c64880fe1be9" containerName="manager"
Sep 15 23:03:42.504934 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.504917    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:03:42.509516 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.509491    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 23:03:42.509666 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.509530    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zbqvm\""
Sep 15 23:03:42.509666 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.509554    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 23:03:42.509666 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.509530    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 23:03:42.513380 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.513353    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54fd547858-fdlgn"]
Sep 15 23:03:42.636946 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.636911    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-metrics-certs\") pod \"maas-controller-54fd547858-fdlgn\" (UID: \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\") " pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:03:42.637139 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.636961    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-webhook-cert\") pod \"maas-controller-54fd547858-fdlgn\" (UID: \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\") " pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:03:42.637139 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.637077    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmwg\" (UniqueName: \"kubernetes.io/projected/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-kube-api-access-xvmwg\") pod \"maas-controller-54fd547858-fdlgn\" (UID: \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\") " pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:03:42.738355 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.738313    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-webhook-cert\") pod \"maas-controller-54fd547858-fdlgn\" (UID: \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\") " pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:03:42.738549 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.738408    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmwg\" (UniqueName: \"kubernetes.io/projected/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-kube-api-access-xvmwg\") pod \"maas-controller-54fd547858-fdlgn\" (UID: \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\") " pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:03:42.738549 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.738472    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-metrics-certs\") pod \"maas-controller-54fd547858-fdlgn\" (UID: \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\") " pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:03:42.740879 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.740850    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-webhook-cert\") pod \"maas-controller-54fd547858-fdlgn\" (UID: \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\") " pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:03:42.740999 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.740909    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-metrics-certs\") pod \"maas-controller-54fd547858-fdlgn\" (UID: \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\") " pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:03:42.747260 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.747235    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmwg\" (UniqueName: \"kubernetes.io/projected/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-kube-api-access-xvmwg\") pod \"maas-controller-54fd547858-fdlgn\" (UID: \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\") " pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:03:42.816307 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:42.816249    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:03:43.088414 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:43.088387    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54fd547858-fdlgn"]
Sep 15 23:03:43.090543 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:03:43.090516    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd979d2f9_c803_49ba_886b_7d6b0ff09a6c.slice/crio-8504c53245451c72480af1103ecc547393655622a0b7924724dd87fb1c901c03 WatchSource:0}: Error finding container 8504c53245451c72480af1103ecc547393655622a0b7924724dd87fb1c901c03: Status 404 returned error can't find the container with id 8504c53245451c72480af1103ecc547393655622a0b7924724dd87fb1c901c03
Sep 15 23:03:43.092236 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:43.092221    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 23:03:43.986896 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:43.986803    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd547858-fdlgn" event={"ID":"d979d2f9-c803-49ba-886b-7d6b0ff09a6c","Type":"ContainerStarted","Data":"c6886361c5eef58aff1eae75aa7898c239d8a5916becfa86cc22815e17967ae8"}
Sep 15 23:03:43.986896 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:43.986856    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd547858-fdlgn" event={"ID":"d979d2f9-c803-49ba-886b-7d6b0ff09a6c","Type":"ContainerStarted","Data":"8504c53245451c72480af1103ecc547393655622a0b7924724dd87fb1c901c03"}
Sep 15 23:03:43.987305 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:43.986901    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:03:44.005133 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:44.005073    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-54fd547858-fdlgn" podStartSLOduration=1.462867228 podStartE2EDuration="2.005055968s" totalImagesPullingTime="542.18874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 23:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 23:03:44.003704526 +0000 UTC m=+2196.622519435" watchObservedRunningTime="2026-09-15 23:03:44.005055968 +0000 UTC m=+2196.623870877"
Sep 15 23:03:54.994851 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:54.994817    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:03:55.406951 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:03:55.406918    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:04:03.758092 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:03.758056    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:04:04.205781 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:04.205747    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:04:04.301688 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:04.301657    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54fd547858-fdlgn"]
Sep 15 23:04:04.301872 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:04.301848    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-54fd547858-fdlgn" podUID="d979d2f9-c803-49ba-886b-7d6b0ff09a6c" containerName="manager" containerID="cri-o://c6886361c5eef58aff1eae75aa7898c239d8a5916becfa86cc22815e17967ae8" gracePeriod=30
Sep 15 23:04:04.573383 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:04.573354    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:04:04.626377 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:04.626341    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvmwg\" (UniqueName: \"kubernetes.io/projected/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-kube-api-access-xvmwg\") pod \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\" (UID: \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\") "
Sep 15 23:04:04.626563 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:04.626420    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-metrics-certs\") pod \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\" (UID: \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\") "
Sep 15 23:04:04.626563 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:04.626467    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-webhook-cert\") pod \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\" (UID: \"d979d2f9-c803-49ba-886b-7d6b0ff09a6c\") "
Sep 15 23:04:04.629107 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:04.629065    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-kube-api-access-xvmwg" (OuterVolumeSpecName: "kube-api-access-xvmwg") pod "d979d2f9-c803-49ba-886b-7d6b0ff09a6c" (UID: "d979d2f9-c803-49ba-886b-7d6b0ff09a6c"). InnerVolumeSpecName "kube-api-access-xvmwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 23:04:04.629348 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:04.629321    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d979d2f9-c803-49ba-886b-7d6b0ff09a6c" (UID: "d979d2f9-c803-49ba-886b-7d6b0ff09a6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 23:04:04.629894 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:04.629871    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d979d2f9-c803-49ba-886b-7d6b0ff09a6c" (UID: "d979d2f9-c803-49ba-886b-7d6b0ff09a6c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 23:04:04.728093 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:04.728016    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvmwg\" (UniqueName: \"kubernetes.io/projected/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-kube-api-access-xvmwg\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:04:04.728093 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:04.728053    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-metrics-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:04:04.728093 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:04.728069    2584 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d979d2f9-c803-49ba-886b-7d6b0ff09a6c-webhook-cert\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:04:05.100561 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.100516    2584 generic.go:356] "Generic (PLEG): container finished" podID="d979d2f9-c803-49ba-886b-7d6b0ff09a6c" containerID="c6886361c5eef58aff1eae75aa7898c239d8a5916becfa86cc22815e17967ae8" exitCode=0
Sep 15 23:04:05.100965 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.100556    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd547858-fdlgn" event={"ID":"d979d2f9-c803-49ba-886b-7d6b0ff09a6c","Type":"ContainerDied","Data":"c6886361c5eef58aff1eae75aa7898c239d8a5916becfa86cc22815e17967ae8"}
Sep 15 23:04:05.100965 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.100612    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd547858-fdlgn" event={"ID":"d979d2f9-c803-49ba-886b-7d6b0ff09a6c","Type":"ContainerDied","Data":"8504c53245451c72480af1103ecc547393655622a0b7924724dd87fb1c901c03"}
Sep 15 23:04:05.100965 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.100633    2584 scope.go:117] "RemoveContainer" containerID="c6886361c5eef58aff1eae75aa7898c239d8a5916becfa86cc22815e17967ae8"
Sep 15 23:04:05.100965 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.100564    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd547858-fdlgn"
Sep 15 23:04:05.110113 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.110096    2584 scope.go:117] "RemoveContainer" containerID="c6886361c5eef58aff1eae75aa7898c239d8a5916becfa86cc22815e17967ae8"
Sep 15 23:04:05.110422 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:04:05.110403    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6886361c5eef58aff1eae75aa7898c239d8a5916becfa86cc22815e17967ae8\": container with ID starting with c6886361c5eef58aff1eae75aa7898c239d8a5916becfa86cc22815e17967ae8 not found: ID does not exist" containerID="c6886361c5eef58aff1eae75aa7898c239d8a5916becfa86cc22815e17967ae8"
Sep 15 23:04:05.110473 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.110431    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6886361c5eef58aff1eae75aa7898c239d8a5916becfa86cc22815e17967ae8"} err="failed to get container status \"c6886361c5eef58aff1eae75aa7898c239d8a5916becfa86cc22815e17967ae8\": rpc error: code = NotFound desc = could not find container \"c6886361c5eef58aff1eae75aa7898c239d8a5916becfa86cc22815e17967ae8\": container with ID starting with c6886361c5eef58aff1eae75aa7898c239d8a5916becfa86cc22815e17967ae8 not found: ID does not exist"
Sep 15 23:04:05.122604 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.122581    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54fd547858-fdlgn"]
Sep 15 23:04:05.125926 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.125903    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-54fd547858-fdlgn"]
Sep 15 23:04:05.783648 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.783615    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:04:05.886571 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.886531    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54fd547858-sg4gn"]
Sep 15 23:04:05.886898 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.886886    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d979d2f9-c803-49ba-886b-7d6b0ff09a6c" containerName="manager"
Sep 15 23:04:05.886943 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.886900    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d979d2f9-c803-49ba-886b-7d6b0ff09a6c" containerName="manager"
Sep 15 23:04:05.886978 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.886957    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d979d2f9-c803-49ba-886b-7d6b0ff09a6c" containerName="manager"
Sep 15 23:04:05.889833 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.889816    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd547858-sg4gn"
Sep 15 23:04:05.894124 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.894101    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 23:04:05.894296 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.894284    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 23:04:05.894372 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.894354    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 23:04:05.894504 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.894487    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zbqvm\""
Sep 15 23:04:05.900552 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.900527    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d979d2f9-c803-49ba-886b-7d6b0ff09a6c" path="/var/lib/kubelet/pods/d979d2f9-c803-49ba-886b-7d6b0ff09a6c/volumes"
Sep 15 23:04:05.900897 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.900879    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54fd547858-sg4gn"]
Sep 15 23:04:05.937535 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.937510    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db4a5367-85b4-484d-9cda-c7ff8d26b96f-webhook-cert\") pod \"maas-controller-54fd547858-sg4gn\" (UID: \"db4a5367-85b4-484d-9cda-c7ff8d26b96f\") " pod="opendatahub/maas-controller-54fd547858-sg4gn"
Sep 15 23:04:05.937639 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.937581    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4a5367-85b4-484d-9cda-c7ff8d26b96f-metrics-certs\") pod \"maas-controller-54fd547858-sg4gn\" (UID: \"db4a5367-85b4-484d-9cda-c7ff8d26b96f\") " pod="opendatahub/maas-controller-54fd547858-sg4gn"
Sep 15 23:04:05.937639 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:05.937608    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvqnx\" (UniqueName: \"kubernetes.io/projected/db4a5367-85b4-484d-9cda-c7ff8d26b96f-kube-api-access-nvqnx\") pod \"maas-controller-54fd547858-sg4gn\" (UID: \"db4a5367-85b4-484d-9cda-c7ff8d26b96f\") " pod="opendatahub/maas-controller-54fd547858-sg4gn"
Sep 15 23:04:06.038410 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:06.038350    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db4a5367-85b4-484d-9cda-c7ff8d26b96f-webhook-cert\") pod \"maas-controller-54fd547858-sg4gn\" (UID: \"db4a5367-85b4-484d-9cda-c7ff8d26b96f\") " pod="opendatahub/maas-controller-54fd547858-sg4gn"
Sep 15 23:04:06.038517 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:06.038456    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4a5367-85b4-484d-9cda-c7ff8d26b96f-metrics-certs\") pod \"maas-controller-54fd547858-sg4gn\" (UID: \"db4a5367-85b4-484d-9cda-c7ff8d26b96f\") " pod="opendatahub/maas-controller-54fd547858-sg4gn"
Sep 15 23:04:06.038517 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:06.038486    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqnx\" (UniqueName: \"kubernetes.io/projected/db4a5367-85b4-484d-9cda-c7ff8d26b96f-kube-api-access-nvqnx\") pod \"maas-controller-54fd547858-sg4gn\" (UID: \"db4a5367-85b4-484d-9cda-c7ff8d26b96f\") " pod="opendatahub/maas-controller-54fd547858-sg4gn"
Sep 15 23:04:06.040803 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:06.040783    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4a5367-85b4-484d-9cda-c7ff8d26b96f-metrics-certs\") pod \"maas-controller-54fd547858-sg4gn\" (UID: \"db4a5367-85b4-484d-9cda-c7ff8d26b96f\") " pod="opendatahub/maas-controller-54fd547858-sg4gn"
Sep 15 23:04:06.040927 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:06.040904    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db4a5367-85b4-484d-9cda-c7ff8d26b96f-webhook-cert\") pod \"maas-controller-54fd547858-sg4gn\" (UID: \"db4a5367-85b4-484d-9cda-c7ff8d26b96f\") " pod="opendatahub/maas-controller-54fd547858-sg4gn"
Sep 15 23:04:06.046171 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:06.046152    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvqnx\" (UniqueName: \"kubernetes.io/projected/db4a5367-85b4-484d-9cda-c7ff8d26b96f-kube-api-access-nvqnx\") pod \"maas-controller-54fd547858-sg4gn\" (UID: \"db4a5367-85b4-484d-9cda-c7ff8d26b96f\") " pod="opendatahub/maas-controller-54fd547858-sg4gn"
Sep 15 23:04:06.202180 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:06.202150    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd547858-sg4gn"
Sep 15 23:04:06.472872 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:06.472845    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54fd547858-sg4gn"]
Sep 15 23:04:06.474401 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:04:06.474371    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4a5367_85b4_484d_9cda_c7ff8d26b96f.slice/crio-c749aac7f52923eae1c67176cb3ab03766784cb22e439776e2e8b8888725fa7a WatchSource:0}: Error finding container c749aac7f52923eae1c67176cb3ab03766784cb22e439776e2e8b8888725fa7a: Status 404 returned error can't find the container with id c749aac7f52923eae1c67176cb3ab03766784cb22e439776e2e8b8888725fa7a
Sep 15 23:04:07.113891 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:07.113855    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd547858-sg4gn" event={"ID":"db4a5367-85b4-484d-9cda-c7ff8d26b96f","Type":"ContainerStarted","Data":"d1b9ee3028040802995b91040827554e994deb7c0eb97bb2d90dce54c2cb7973"}
Sep 15 23:04:07.114069 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:07.113912    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd547858-sg4gn" event={"ID":"db4a5367-85b4-484d-9cda-c7ff8d26b96f","Type":"ContainerStarted","Data":"c749aac7f52923eae1c67176cb3ab03766784cb22e439776e2e8b8888725fa7a"}
Sep 15 23:04:07.114069 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:07.113993    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54fd547858-sg4gn"
Sep 15 23:04:07.131472 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:07.131378    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-54fd547858-sg4gn" podStartSLOduration=1.73522815 podStartE2EDuration="2.131359992s" totalImagesPullingTime="396.131842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 23:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 23:04:07.130777834 +0000 UTC m=+2219.749592742" watchObservedRunningTime="2026-09-15 23:04:07.131359992 +0000 UTC m=+2219.750174931"
Sep 15 23:04:18.120800 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:18.120763    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54fd547858-sg4gn"
Sep 15 23:04:32.069485 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:32.069404    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:04:39.866922 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:39.866879    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:04:47.068358 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:47.068324    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:04:54.359372 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:04:54.359335    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:05:08.869704 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:05:08.869661    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:05:16.854692 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:05:16.854655    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:05:34.876921 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:05:34.876878    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:06:08.978161 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:06:08.978084    2584 scope.go:117] "RemoveContainer" containerID="c026772b2d5503c23f27c146f94badd0ebe6457452b7c98efa6a3df7bbaa4b14"
Sep 15 23:06:08.986331 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:06:08.986308    2584 scope.go:117] "RemoveContainer" containerID="d28602cf1fe6bf3b3c94ac2abddb0a0ace938cf41c23b61b9fa497ba1ea7f0b2"
Sep 15 23:06:45.857499 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:06:45.857465    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:06:52.662635 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:06:52.662604    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:07:02.056098 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:02.056063    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:07:07.456572 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.456539    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"]
Sep 15 23:07:07.460104 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.460081    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.462935 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.462914    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6f420dbe-kserve-self-signed-certs\""
Sep 15 23:07:07.470717 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.470692    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"]
Sep 15 23:07:07.517379 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.517348    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:07:07.565549 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.565521    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-home\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.565664 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.565552    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-tmp-dir\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.565664 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.565596    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-dshm\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.565664 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.565620    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-model-cache\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.565664 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.565641    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6n2t\" (UniqueName: \"kubernetes.io/projected/c82d16d5-8ca4-47e4-b670-4228459e929d-kube-api-access-b6n2t\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.565805 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.565686    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c82d16d5-8ca4-47e4-b670-4228459e929d-tls-certs\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.667024 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.666984    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-dshm\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.667213 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.667034    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-model-cache\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.667213 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.667058    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6n2t\" (UniqueName: \"kubernetes.io/projected/c82d16d5-8ca4-47e4-b670-4228459e929d-kube-api-access-b6n2t\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.667213 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.667090    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c82d16d5-8ca4-47e4-b670-4228459e929d-tls-certs\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.667426 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.667217    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-home\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.667426 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.667281    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-tmp-dir\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.667577 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.667549    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-model-cache\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.667708 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.667602    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-home\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.667708 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.667621    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-tmp-dir\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.669384 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.669361    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-dshm\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.669627 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.669612    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c82d16d5-8ca4-47e4-b670-4228459e929d-tls-certs\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.676247 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.676221    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6n2t\" (UniqueName: \"kubernetes.io/projected/c82d16d5-8ca4-47e4-b670-4228459e929d-kube-api-access-b6n2t\") pod \"e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") " pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:07.771629 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:07.771554    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:08.044387 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:08.044354    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"]
Sep 15 23:07:08.046097 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:07:08.046058    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc82d16d5_8ca4_47e4_b670_4228459e929d.slice/crio-5169c4c751569166c14c338e7d8f3395f80e6c99065d8201b727e31ddeb7899c WatchSource:0}: Error finding container 5169c4c751569166c14c338e7d8f3395f80e6c99065d8201b727e31ddeb7899c: Status 404 returned error can't find the container with id 5169c4c751569166c14c338e7d8f3395f80e6c99065d8201b727e31ddeb7899c
Sep 15 23:07:08.055207 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:08.055191    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 23:07:08.060805 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:08.060776    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 23:07:09.008777 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:09.008748    2584 scope.go:117] "RemoveContainer" containerID="7de742a0cbfcce69c32035ed1ca921ed7b250f6aba63a09bdaee6fec89c696f6"
Sep 15 23:07:09.016911 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:09.016894    2584 scope.go:117] "RemoveContainer" containerID="6906d2759794e309154961980ee9dd86d8abd22948f8fe6b65e93226df56da50"
Sep 15 23:07:09.025808 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:09.025785    2584 scope.go:117] "RemoveContainer" containerID="9690ad433905029f494e9bb002b3fba1e645b847700343871a6bc1c3b21718a6"
Sep 15 23:07:09.037597 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:09.037565    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw" event={"ID":"c82d16d5-8ca4-47e4-b670-4228459e929d","Type":"ContainerStarted","Data":"d1ed91ab2363ce0aa54abda86bd35874bfb87a0620b0087887099a11b5e3a1ad"}
Sep 15 23:07:09.037715 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:09.037615    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw" event={"ID":"c82d16d5-8ca4-47e4-b670-4228459e929d","Type":"ContainerStarted","Data":"5169c4c751569166c14c338e7d8f3395f80e6c99065d8201b727e31ddeb7899c"}
Sep 15 23:07:09.055763 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:09.055720    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw" podStartSLOduration=2.055706907 podStartE2EDuration="2.055706907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 23:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 23:07:09.054370705 +0000 UTC m=+2401.673185615" watchObservedRunningTime="2026-09-15 23:07:09.055706907 +0000 UTC m=+2401.674521814"
Sep 15 23:07:09.865967 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:09.865928    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:07:11.704150 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:11.704109    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:07:17.772459 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:17.772422    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:17.773007 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:17.772714    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:17.785486 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:17.785462    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:18.095188 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:18.095159    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:07:43.371616 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:43.371553    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:07:46.365246 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:07:46.365209    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:08:06.536786 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:06.536749    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"]
Sep 15 23:08:06.555381 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:06.555351    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:08:08.258005 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:08.257972    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:08:08.868863 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:08.868831    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:08:09.052175 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:09.052146    2584 scope.go:117] "RemoveContainer" containerID="696b8f833ca6089946c206ac0cf6f0a3d73aad3135a640991b361302f89ff595"
Sep 15 23:08:09.060584 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:09.060559    2584 scope.go:117] "RemoveContainer" containerID="70c91acb3e036717e9e38beeb18d20a0dff0f1b3f034808743e0ac44434b3834"
Sep 15 23:08:17.564339 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:17.564303    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:08:21.556708 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.556663    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw" podUID="c82d16d5-8ca4-47e4-b670-4228459e929d" containerName="main" containerID="cri-o://d1ed91ab2363ce0aa54abda86bd35874bfb87a0620b0087887099a11b5e3a1ad" gracePeriod=45
Sep 15 23:08:21.805146 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.805122    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:08:21.927480 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.927394    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6n2t\" (UniqueName: \"kubernetes.io/projected/c82d16d5-8ca4-47e4-b670-4228459e929d-kube-api-access-b6n2t\") pod \"c82d16d5-8ca4-47e4-b670-4228459e929d\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") "
Sep 15 23:08:21.927480 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.927452    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-dshm\") pod \"c82d16d5-8ca4-47e4-b670-4228459e929d\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") "
Sep 15 23:08:21.927716 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.927503    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c82d16d5-8ca4-47e4-b670-4228459e929d-tls-certs\") pod \"c82d16d5-8ca4-47e4-b670-4228459e929d\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") "
Sep 15 23:08:21.927716 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.927528    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-tmp-dir\") pod \"c82d16d5-8ca4-47e4-b670-4228459e929d\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") "
Sep 15 23:08:21.927716 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.927575    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-home\") pod \"c82d16d5-8ca4-47e4-b670-4228459e929d\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") "
Sep 15 23:08:21.927716 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.927606    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-model-cache\") pod \"c82d16d5-8ca4-47e4-b670-4228459e929d\" (UID: \"c82d16d5-8ca4-47e4-b670-4228459e929d\") "
Sep 15 23:08:21.927936 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.927877    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c82d16d5-8ca4-47e4-b670-4228459e929d" (UID: "c82d16d5-8ca4-47e4-b670-4228459e929d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:08:21.927936 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.927886    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-home" (OuterVolumeSpecName: "home") pod "c82d16d5-8ca4-47e4-b670-4228459e929d" (UID: "c82d16d5-8ca4-47e4-b670-4228459e929d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:08:21.927936 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.927916    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c82d16d5-8ca4-47e4-b670-4228459e929d" (UID: "c82d16d5-8ca4-47e4-b670-4228459e929d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:08:21.929722 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.929695    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-dshm" (OuterVolumeSpecName: "dshm") pod "c82d16d5-8ca4-47e4-b670-4228459e929d" (UID: "c82d16d5-8ca4-47e4-b670-4228459e929d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 23:08:21.929831 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.929728    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82d16d5-8ca4-47e4-b670-4228459e929d-kube-api-access-b6n2t" (OuterVolumeSpecName: "kube-api-access-b6n2t") pod "c82d16d5-8ca4-47e4-b670-4228459e929d" (UID: "c82d16d5-8ca4-47e4-b670-4228459e929d"). InnerVolumeSpecName "kube-api-access-b6n2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 23:08:21.929831 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:21.929758    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82d16d5-8ca4-47e4-b670-4228459e929d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c82d16d5-8ca4-47e4-b670-4228459e929d" (UID: "c82d16d5-8ca4-47e4-b670-4228459e929d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 23:08:22.029440 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.029403    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6n2t\" (UniqueName: \"kubernetes.io/projected/c82d16d5-8ca4-47e4-b670-4228459e929d-kube-api-access-b6n2t\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:08:22.029440 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.029442    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-dshm\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:08:22.029628 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.029457    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c82d16d5-8ca4-47e4-b670-4228459e929d-tls-certs\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:08:22.029628 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.029471    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-tmp-dir\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:08:22.029628 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.029483    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-home\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:08:22.029628 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.029492    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c82d16d5-8ca4-47e4-b670-4228459e929d-model-cache\") on node \"ip-10-0-138-116.ec2.internal\" DevicePath \"\""
Sep 15 23:08:22.414724 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.414684    2584 generic.go:356] "Generic (PLEG): container finished" podID="c82d16d5-8ca4-47e4-b670-4228459e929d" containerID="d1ed91ab2363ce0aa54abda86bd35874bfb87a0620b0087887099a11b5e3a1ad" exitCode=0
Sep 15 23:08:22.414724 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.414726    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"
Sep 15 23:08:22.414959 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.414768    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw" event={"ID":"c82d16d5-8ca4-47e4-b670-4228459e929d","Type":"ContainerDied","Data":"d1ed91ab2363ce0aa54abda86bd35874bfb87a0620b0087887099a11b5e3a1ad"}
Sep 15 23:08:22.414959 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.414811    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw" event={"ID":"c82d16d5-8ca4-47e4-b670-4228459e929d","Type":"ContainerDied","Data":"5169c4c751569166c14c338e7d8f3395f80e6c99065d8201b727e31ddeb7899c"}
Sep 15 23:08:22.414959 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.414828    2584 scope.go:117] "RemoveContainer" containerID="d1ed91ab2363ce0aa54abda86bd35874bfb87a0620b0087887099a11b5e3a1ad"
Sep 15 23:08:22.423859 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.423842    2584 scope.go:117] "RemoveContainer" containerID="d1ed91ab2363ce0aa54abda86bd35874bfb87a0620b0087887099a11b5e3a1ad"
Sep 15 23:08:22.424145 ip-10-0-138-116 kubenswrapper[2584]: E0915 23:08:22.424124    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ed91ab2363ce0aa54abda86bd35874bfb87a0620b0087887099a11b5e3a1ad\": container with ID starting with d1ed91ab2363ce0aa54abda86bd35874bfb87a0620b0087887099a11b5e3a1ad not found: ID does not exist" containerID="d1ed91ab2363ce0aa54abda86bd35874bfb87a0620b0087887099a11b5e3a1ad"
Sep 15 23:08:22.424193 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.424155    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ed91ab2363ce0aa54abda86bd35874bfb87a0620b0087887099a11b5e3a1ad"} err="failed to get container status \"d1ed91ab2363ce0aa54abda86bd35874bfb87a0620b0087887099a11b5e3a1ad\": rpc error: code = NotFound desc = could not find container \"d1ed91ab2363ce0aa54abda86bd35874bfb87a0620b0087887099a11b5e3a1ad\": container with ID starting with d1ed91ab2363ce0aa54abda86bd35874bfb87a0620b0087887099a11b5e3a1ad not found: ID does not exist"
Sep 15 23:08:22.437097 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.437076    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"]
Sep 15 23:08:22.445364 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:22.445343    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6f420dbe-kserve-85c97c8787-5hxnw"]
Sep 15 23:08:23.899052 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:23.899019    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c82d16d5-8ca4-47e4-b670-4228459e929d" path="/var/lib/kubelet/pods/c82d16d5-8ca4-47e4-b670-4228459e929d/volumes"
Sep 15 23:08:25.155470 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:25.155440    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:08:31.375077 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:31.375039    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:08:38.677122 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:38.677083    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:08:45.957230 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:45.957192    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:08:53.861383 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:53.861349    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:08:59.470346 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:08:59.470311    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ql74p"]
Sep 15 23:09:07.567643 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:07.567613    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-54fd547858-sg4gn_db4a5367-85b4-484d-9cda-c7ff8d26b96f/manager/0.log"
Sep 15 23:09:07.927190 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:07.927119    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749d6986f8-88k9d_0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d/copy-manifests/0.log"
Sep 15 23:09:07.938902 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:07.938876    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749d6986f8-88k9d_0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d/manager/0.log"
Sep 15 23:09:08.827284 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:08.827252    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825220-2t7mh_a2210ea7-1cd5-454f-ba06-ff594a74adc1/cleanup/0.log"
Sep 15 23:09:09.082416 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:09.082327    2584 scope.go:117] "RemoveContainer" containerID="8bb4c48bffde11ad41a59e861f88a1c7f481f8866ab8f11a684e9da509377b09"
Sep 15 23:09:09.773159 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:09.773128    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:09:09.945720 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:09.945688    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vhj44_0788b7e2-df30-4fe8-9f0e-c08beded8818/manager/0.log"
Sep 15 23:09:10.308664 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:10.308614    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-ql74p_f5a40e7f-6cff-4a38-a3a0-ce490ff37330/limitador/0.log"
Sep 15 23:09:11.216816 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:11.216783    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8b2xb_847f9278-fbd9-44ff-b475-62af5997177f/istio-proxy/1.log"
Sep 15 23:09:11.441600 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:11.441565    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x6hfn_af0f1d9d-0d60-4861-85c6-e991c8e58bd7/payload-processing/0.log"
Sep 15 23:09:11.550496 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:11.550469    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6df89d9594-wrhv2_f53b8e84-9062-4265-9b5e-4d4b40ad7020/router/0.log"
Sep 15 23:09:11.882095 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:11.882018    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-xqk2t_5ecc93c7-3ce3-4f8b-ad13-2966714ac9f3/main/0.log"
Sep 15 23:09:12.100428 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:12.100397    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-d5vq9_4babe645-19bd-4280-8420-9084fe9858ce/main/0.log"
Sep 15 23:09:12.536668 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:12.536639    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-fm5gz_280a414e-4228-48e1-881a-7dd9c5358afd/main/0.log"
Sep 15 23:09:17.497879 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.497844    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ctgsf/must-gather-jcmn5"]
Sep 15 23:09:17.498330 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.498205    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82d16d5-8ca4-47e4-b670-4228459e929d" containerName="main"
Sep 15 23:09:17.498330 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.498217    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82d16d5-8ca4-47e4-b670-4228459e929d" containerName="main"
Sep 15 23:09:17.498330 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.498283    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82d16d5-8ca4-47e4-b670-4228459e929d" containerName="main"
Sep 15 23:09:17.501315 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.501285    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ctgsf/must-gather-jcmn5"
Sep 15 23:09:17.504014 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.503994    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ctgsf\"/\"default-dockercfg-drr56\""
Sep 15 23:09:17.504138 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.504103    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ctgsf\"/\"kube-root-ca.crt\""
Sep 15 23:09:17.504214 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.504138    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ctgsf\"/\"openshift-service-ca.crt\""
Sep 15 23:09:17.512764 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.512739    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ctgsf/must-gather-jcmn5"]
Sep 15 23:09:17.515701 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.515679    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:09:17.602439 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.602417    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bfzp\" (UniqueName: \"kubernetes.io/projected/972861af-2418-4e9f-86f1-242af5106eb9-kube-api-access-8bfzp\") pod \"must-gather-jcmn5\" (UID: \"972861af-2418-4e9f-86f1-242af5106eb9\") " pod="openshift-must-gather-ctgsf/must-gather-jcmn5"
Sep 15 23:09:17.602547 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.602446    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/972861af-2418-4e9f-86f1-242af5106eb9-must-gather-output\") pod \"must-gather-jcmn5\" (UID: \"972861af-2418-4e9f-86f1-242af5106eb9\") " pod="openshift-must-gather-ctgsf/must-gather-jcmn5"
Sep 15 23:09:17.618807 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.618786    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log" limit=1024
Sep 15 23:09:17.703504 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.703481    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bfzp\" (UniqueName: \"kubernetes.io/projected/972861af-2418-4e9f-86f1-242af5106eb9-kube-api-access-8bfzp\") pod \"must-gather-jcmn5\" (UID: \"972861af-2418-4e9f-86f1-242af5106eb9\") " pod="openshift-must-gather-ctgsf/must-gather-jcmn5"
Sep 15 23:09:17.703604 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.703512    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/972861af-2418-4e9f-86f1-242af5106eb9-must-gather-output\") pod \"must-gather-jcmn5\" (UID: \"972861af-2418-4e9f-86f1-242af5106eb9\") " pod="openshift-must-gather-ctgsf/must-gather-jcmn5"
Sep 15 23:09:17.703776 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.703763    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/972861af-2418-4e9f-86f1-242af5106eb9-must-gather-output\") pod \"must-gather-jcmn5\" (UID: \"972861af-2418-4e9f-86f1-242af5106eb9\") " pod="openshift-must-gather-ctgsf/must-gather-jcmn5"
Sep 15 23:09:17.712475 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.712457    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bfzp\" (UniqueName: \"kubernetes.io/projected/972861af-2418-4e9f-86f1-242af5106eb9-kube-api-access-8bfzp\") pod \"must-gather-jcmn5\" (UID: \"972861af-2418-4e9f-86f1-242af5106eb9\") " pod="openshift-must-gather-ctgsf/must-gather-jcmn5"
Sep 15 23:09:17.812076 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:17.812045    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ctgsf/must-gather-jcmn5"
Sep 15 23:09:18.082174 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:18.082149    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ctgsf/must-gather-jcmn5"]
Sep 15 23:09:18.084470 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:09:18.084441    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod972861af_2418_4e9f_86f1_242af5106eb9.slice/crio-0ad658e7f9c852a47316146841aa9a3a3c920b2779f0d40227bc50e9af15e013 WatchSource:0}: Error finding container 0ad658e7f9c852a47316146841aa9a3a3c920b2779f0d40227bc50e9af15e013: Status 404 returned error can't find the container with id 0ad658e7f9c852a47316146841aa9a3a3c920b2779f0d40227bc50e9af15e013
Sep 15 23:09:18.086191 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:18.086173    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 23:09:18.685085 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:18.685049    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ctgsf/must-gather-jcmn5" event={"ID":"972861af-2418-4e9f-86f1-242af5106eb9","Type":"ContainerStarted","Data":"0ad658e7f9c852a47316146841aa9a3a3c920b2779f0d40227bc50e9af15e013"}
Sep 15 23:09:19.692660 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:19.692615    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ctgsf/must-gather-jcmn5" event={"ID":"972861af-2418-4e9f-86f1-242af5106eb9","Type":"ContainerStarted","Data":"fcb67d5b4a34c910aae00e73fac0ac0bd41e5f115d19d2b0512337d4c093ee12"}
Sep 15 23:09:19.693030 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:19.692669    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ctgsf/must-gather-jcmn5" event={"ID":"972861af-2418-4e9f-86f1-242af5106eb9","Type":"ContainerStarted","Data":"bdd22e10ae9327ca73cc09edfc1274fe0fb0737c575f042e6583014cc39d6109"}
Sep 15 23:09:19.711611 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:19.711550    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ctgsf/must-gather-jcmn5" podStartSLOduration=2.075945728 podStartE2EDuration="2.711531103s" totalImagesPullingTime="635.585375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 23:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 23:09:19.709978581 +0000 UTC m=+2532.328793489" watchObservedRunningTime="2026-09-15 23:09:19.711531103 +0000 UTC m=+2532.330346012"
Sep 15 23:09:20.381366 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:20.381331    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mn2ls_7f0b1be9-4151-4e3e-9ca2-e9aeffdf050c/global-pull-secret-syncer/0.log"
Sep 15 23:09:20.435819 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:20.435775    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5nc6l_38aa201f-97b9-443e-b246-dd19d606eb43/konnectivity-agent/0.log"
Sep 15 23:09:20.551158 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:20.551115    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-116.ec2.internal_69eda65fcd513f716093f5dd7f2f0e50/haproxy/0.log"
Sep 15 23:09:25.429477 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:25.429442    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vhj44_0788b7e2-df30-4fe8-9f0e-c08beded8818/manager/0.log"
Sep 15 23:09:25.529796 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:25.529760    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-ql74p_f5a40e7f-6cff-4a38-a3a0-ce490ff37330/limitador/0.log"
Sep 15 23:09:27.182747 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:27.182710    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-gx49h_67b271e9-a2b8-4f33-be0b-102e434588ac/cluster-monitoring-operator/0.log"
Sep 15 23:09:27.397183 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:27.397156    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cdr4b_30be65c9-2696-450f-831a-f05c587edcfa/node-exporter/0.log"
Sep 15 23:09:27.419956 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:27.419927    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cdr4b_30be65c9-2696-450f-831a-f05c587edcfa/kube-rbac-proxy/0.log"
Sep 15 23:09:27.448240 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:27.448163    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cdr4b_30be65c9-2696-450f-831a-f05c587edcfa/init-textfile/0.log"
Sep 15 23:09:27.851080 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:27.851050    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-cjtlt_3b2a046f-4acd-42ed-825e-3e5b8a69ebc5/prometheus-operator-admission-webhook/0.log"
Sep 15 23:09:29.502001 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:29.501966    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-gkrjm_1f19bd34-abe0-4e3d-9245-6d577ed14679/networking-console-plugin/0.log"
Sep 15 23:09:30.111681 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.111650    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/1.log"
Sep 15 23:09:30.118465 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.118436    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9xx4f_b68f8430-b1b6-46d9-8358-74f04b66c9d6/console-operator/2.log"
Sep 15 23:09:30.367977 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.367891    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"]
Sep 15 23:09:30.373943 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.373918    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.383641 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.383555    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"]
Sep 15 23:09:30.433237 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.433199    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3018207d-d739-40a1-b540-6864d505a6ca-podres\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.433237 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.433249    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3018207d-d739-40a1-b540-6864d505a6ca-proc\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.433520 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.433385    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3018207d-d739-40a1-b540-6864d505a6ca-sys\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.433520 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.433438    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3018207d-d739-40a1-b540-6864d505a6ca-lib-modules\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.433634 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.433521    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl7dq\" (UniqueName: \"kubernetes.io/projected/3018207d-d739-40a1-b540-6864d505a6ca-kube-api-access-zl7dq\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.534735 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.534695    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3018207d-d739-40a1-b540-6864d505a6ca-lib-modules\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.535297 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.535016    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3018207d-d739-40a1-b540-6864d505a6ca-lib-modules\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.535426 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.535340    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl7dq\" (UniqueName: \"kubernetes.io/projected/3018207d-d739-40a1-b540-6864d505a6ca-kube-api-access-zl7dq\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.535552 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.535483    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3018207d-d739-40a1-b540-6864d505a6ca-podres\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.535552 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.535525    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3018207d-d739-40a1-b540-6864d505a6ca-proc\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.535662 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.535630    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3018207d-d739-40a1-b540-6864d505a6ca-sys\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.535716 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.535661    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3018207d-d739-40a1-b540-6864d505a6ca-podres\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.535764 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.535750    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3018207d-d739-40a1-b540-6864d505a6ca-proc\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.535819 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.535793    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3018207d-d739-40a1-b540-6864d505a6ca-sys\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.544369 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.544345    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl7dq\" (UniqueName: \"kubernetes.io/projected/3018207d-d739-40a1-b540-6864d505a6ca-kube-api-access-zl7dq\") pod \"perf-node-gather-daemonset-r6d7j\" (UID: \"3018207d-d739-40a1-b540-6864d505a6ca\") " pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.688695 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.688413    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:30.723018 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.722990    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-6tfr7_85c6151f-5846-4b92-b686-4a4efcbcc1ce/download-server/0.log"
Sep 15 23:09:30.985634 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:30.985600    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"]
Sep 15 23:09:30.988870 ip-10-0-138-116 kubenswrapper[2584]: W0915 23:09:30.988413    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3018207d_d739_40a1_b540_6864d505a6ca.slice/crio-4b766093d76feb7e4b77046797738ff24aeda16c0c953b577fd328062ba88600 WatchSource:0}: Error finding container 4b766093d76feb7e4b77046797738ff24aeda16c0c953b577fd328062ba88600: Status 404 returned error can't find the container with id 4b766093d76feb7e4b77046797738ff24aeda16c0c953b577fd328062ba88600
Sep 15 23:09:31.410629 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:31.410599    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-5kgc5_7f6ab2a4-b56d-45b9-81d8-1319abade096/volume-data-source-validator/0.log"
Sep 15 23:09:31.808326 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:31.808283    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j" event={"ID":"3018207d-d739-40a1-b540-6864d505a6ca","Type":"ContainerStarted","Data":"f21ed663e1c90124614248387d7917dcfd95da469b14124196cd74fab852969e"}
Sep 15 23:09:31.808807 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:31.808355    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j" event={"ID":"3018207d-d739-40a1-b540-6864d505a6ca","Type":"ContainerStarted","Data":"4b766093d76feb7e4b77046797738ff24aeda16c0c953b577fd328062ba88600"}
Sep 15 23:09:31.809151 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:31.809123    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:31.825382 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:31.825327    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j" podStartSLOduration=1.825307174 podStartE2EDuration="1.825307174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 23:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 23:09:31.824685753 +0000 UTC m=+2544.443500664" watchObservedRunningTime="2026-09-15 23:09:31.825307174 +0000 UTC m=+2544.444122084"
Sep 15 23:09:32.626607 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:32.626574    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sm96j_0eea82c0-60f7-4ee7-9a1d-8bd13f06c028/dns/0.log"
Sep 15 23:09:32.646417 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:32.646392    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sm96j_0eea82c0-60f7-4ee7-9a1d-8bd13f06c028/kube-rbac-proxy/0.log"
Sep 15 23:09:32.668309 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:32.668279    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dwfjr_e3deae5a-e1be-4af6-8ac0-ef4976250510/dns-node-resolver/0.log"
Sep 15 23:09:33.277694 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:33.277665    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8jwbj_c8e67238-6768-4b3c-89f2-a17e61edee60/node-ca/0.log"
Sep 15 23:09:34.716886 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:34.716852    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8b2xb_847f9278-fbd9-44ff-b475-62af5997177f/istio-proxy/0.log"
Sep 15 23:09:34.720366 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:34.720339    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8b2xb_847f9278-fbd9-44ff-b475-62af5997177f/istio-proxy/1.log"
Sep 15 23:09:34.779471 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:34.779441    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x6hfn_af0f1d9d-0d60-4861-85c6-e991c8e58bd7/payload-processing/0.log"
Sep 15 23:09:34.806630 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:34.806603    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6df89d9594-wrhv2_f53b8e84-9062-4265-9b5e-4d4b40ad7020/router/0.log"
Sep 15 23:09:35.684313 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:35.684253    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bw662_4c4b4d38-64d6-408a-bf2f-591822b41ac2/serve-healthcheck-canary/0.log"
Sep 15 23:09:36.189708 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:36.189674    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9x825_fc2ba350-9c05-4b47-9f16-8b19093b9137/insights-operator/0.log"
Sep 15 23:09:36.190131 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:36.189908    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9x825_fc2ba350-9c05-4b47-9f16-8b19093b9137/insights-operator/1.log"
Sep 15 23:09:36.212890 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:36.212864    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b46hw_17be5533-4f9e-4bd3-9198-48f1cb3e8ba8/kube-rbac-proxy/0.log"
Sep 15 23:09:36.232320 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:36.232296    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b46hw_17be5533-4f9e-4bd3-9198-48f1cb3e8ba8/exporter/0.log"
Sep 15 23:09:36.251958 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:36.251914    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b46hw_17be5533-4f9e-4bd3-9198-48f1cb3e8ba8/extractor/0.log"
Sep 15 23:09:38.832331 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:38.832300    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ctgsf/perf-node-gather-daemonset-r6d7j"
Sep 15 23:09:38.865891 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:38.865861    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-54fd547858-sg4gn_db4a5367-85b4-484d-9cda-c7ff8d26b96f/manager/0.log"
Sep 15 23:09:39.058954 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:39.058923    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749d6986f8-88k9d_0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d/manager/0.log"
Sep 15 23:09:39.076373 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:39.076341    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749d6986f8-88k9d_0a83cd6d-f5f3-4bd4-ae82-a1d6c06c288d/copy-manifests/0.log"
Sep 15 23:09:40.474942 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:40.474909    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7bd6c5dcdd-tzzhw_8694a0ff-df44-40a4-8b6e-5386c170a94f/manager/0.log"
Sep 15 23:09:40.961177 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:40.961094    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8b2xb_847f9278-fbd9-44ff-b475-62af5997177f/istio-proxy/0.log"
Sep 15 23:09:40.964229 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:40.964196    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8b2xb_847f9278-fbd9-44ff-b475-62af5997177f/istio-proxy/1.log"
Sep 15 23:09:41.009017 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:41.008988    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x6hfn_af0f1d9d-0d60-4861-85c6-e991c8e58bd7/payload-processing/0.log"
Sep 15 23:09:41.024903 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:41.024873    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6df89d9594-wrhv2_f53b8e84-9062-4265-9b5e-4d4b40ad7020/router/0.log"
Sep 15 23:09:42.993681 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:42.993640    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8b2xb_847f9278-fbd9-44ff-b475-62af5997177f/istio-proxy/0.log"
Sep 15 23:09:42.998617 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:42.998578    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8b2xb_847f9278-fbd9-44ff-b475-62af5997177f/istio-proxy/1.log"
Sep 15 23:09:43.046979 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:43.046947    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x6hfn_af0f1d9d-0d60-4861-85c6-e991c8e58bd7/payload-processing/0.log"
Sep 15 23:09:43.068077 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:43.068048    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6df89d9594-wrhv2_f53b8e84-9062-4265-9b5e-4d4b40ad7020/router/0.log"
Sep 15 23:09:45.613220 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:45.613190    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-xdpxh_34d78629-cec9-4ed2-a073-5152edc9dec7/migrator/0.log"
Sep 15 23:09:45.631813 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:45.631790    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-xdpxh_34d78629-cec9-4ed2-a073-5152edc9dec7/graceful-termination/0.log"
Sep 15 23:09:45.996441 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:45.996367    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-qc5w9_702fee44-5a8b-4364-a00f-957fab8661dd/kube-storage-version-migrator-operator/1.log"
Sep 15 23:09:45.997119 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:45.997098    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-qc5w9_702fee44-5a8b-4364-a00f-957fab8661dd/kube-storage-version-migrator-operator/0.log"
Sep 15 23:09:47.038561 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:47.038532    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fz5x_86c5e1a3-16e0-4171-a9a2-df5f8eba5318/kube-multus-additional-cni-plugins/0.log"
Sep 15 23:09:47.060587 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:47.060560    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fz5x_86c5e1a3-16e0-4171-a9a2-df5f8eba5318/egress-router-binary-copy/0.log"
Sep 15 23:09:47.082758 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:47.082734    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fz5x_86c5e1a3-16e0-4171-a9a2-df5f8eba5318/cni-plugins/0.log"
Sep 15 23:09:47.103757 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:47.103738    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fz5x_86c5e1a3-16e0-4171-a9a2-df5f8eba5318/bond-cni-plugin/0.log"
Sep 15 23:09:47.123608 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:47.123587    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fz5x_86c5e1a3-16e0-4171-a9a2-df5f8eba5318/routeoverride-cni/0.log"
Sep 15 23:09:47.143098 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:47.143079    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fz5x_86c5e1a3-16e0-4171-a9a2-df5f8eba5318/whereabouts-cni-bincopy/0.log"
Sep 15 23:09:47.161643 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:47.161622    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fz5x_86c5e1a3-16e0-4171-a9a2-df5f8eba5318/whereabouts-cni/0.log"
Sep 15 23:09:47.501679 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:47.501648    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-llx9t_c298fcdd-c5f5-4410-995a-8d66952752b8/kube-multus/0.log"
Sep 15 23:09:47.638601 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:47.638562    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pc4cx_e8d2b3b9-6751-4c72-9677-dabc86c07f2a/network-metrics-daemon/0.log"
Sep 15 23:09:47.654092 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:47.654058    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pc4cx_e8d2b3b9-6751-4c72-9677-dabc86c07f2a/kube-rbac-proxy/0.log"
Sep 15 23:09:48.722238 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:48.722210    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hd5sn_5e931f99-26f8-440d-a9b9-269517778506/ovn-controller/0.log"
Sep 15 23:09:48.737993 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:48.737968    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hd5sn_5e931f99-26f8-440d-a9b9-269517778506/ovn-acl-logging/0.log"
Sep 15 23:09:48.762863 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:48.762836    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hd5sn_5e931f99-26f8-440d-a9b9-269517778506/kube-rbac-proxy-node/0.log"
Sep 15 23:09:48.785077 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:48.785051    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hd5sn_5e931f99-26f8-440d-a9b9-269517778506/kube-rbac-proxy-ovn-metrics/0.log"
Sep 15 23:09:48.804226 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:48.804197    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hd5sn_5e931f99-26f8-440d-a9b9-269517778506/northd/0.log"
Sep 15 23:09:48.824743 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:48.824722    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hd5sn_5e931f99-26f8-440d-a9b9-269517778506/nbdb/0.log"
Sep 15 23:09:48.846119 ip-10-0-138-116 kubenswrapper[2584]: I0915 23:09:48.846099    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hd5sn_5e931f99-26f8-440d-a9b9-269517778506/sbdb/0.log"