Sep 18 15:32:21.065380 ip-10-0-130-223 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 15:32:21.065393 ip-10-0-130-223 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 15:32:21.065403 ip-10-0-130-223 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 15:32:21.065603 ip-10-0-130-223 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 15:32:31.311448 ip-10-0-130-223 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 15:32:31.311467 ip-10-0-130-223 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2af57871470945a5b8a3c559de907b8d --
Sep 18 15:34:35.974130 ip-10-0-130-223 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 15:34:39.507642 ip-10-0-130-223 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 15:34:39.507642 ip-10-0-130-223 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 15:34:39.507642 ip-10-0-130-223 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 15:34:39.507642 ip-10-0-130-223 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 15:34:39.507642 ip-10-0-130-223 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 15:34:39.508491 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.507712    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 15:34:39.510720 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510702    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510722    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510725    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510728    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510731    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510734    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510737    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510741    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510744    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510746    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510749    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510752    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510755    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510757    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510760    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510763    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510770    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 15:34:39.510765 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510773    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510776    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510779    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510781    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510784    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510787    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510790    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510792    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510797    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510801    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510804    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510806    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510809    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510812    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510815    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510818    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510821    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510823    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510826    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510828    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 15:34:39.511197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510831    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510833    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510836    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510840    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510843    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510846    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510848    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510864    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510867    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510869    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510872    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510876    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510878    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510881    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510884    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510887    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510892    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510894    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510897    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 15:34:39.511682 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510899    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510903    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510905    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510908    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510910    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510913    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510916    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510918    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510921    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510923    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510926    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510928    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510931    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510933    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510936    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510939    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510942    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510945    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510948    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510951    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 15:34:39.512215 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510953    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510956    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510959    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510963    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510965    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510968    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510971    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510976    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510981    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510983    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510986    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510989    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510992    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510994    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510997    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.510999    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.511002    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.511004    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.511007    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.511009    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 15:34:39.512697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.511012    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.511015    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.511017    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512312    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512321    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512324    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512327    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512330    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512333    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512337    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512340    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512342    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512345    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512348    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512351    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512354    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512357    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512361    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512366    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512369    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 15:34:39.513199 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512372    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512374    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512377    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512380    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512383    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512386    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512389    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512391    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512393    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512396    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512399    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512401    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512404    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512406    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512409    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512412    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512415    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512417    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512420    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512422    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 15:34:39.513737 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512425    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512428    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512430    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512433    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512435    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512438    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512441    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512444    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512446    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512450    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512453    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512456    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512459    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512461    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512464    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512467    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512469    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512471    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512474    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512477    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 15:34:39.514264 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512479    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512482    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512484    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512487    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512489    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512492    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512494    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512497    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512499    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512502    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512504    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512507    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512510    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512512    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512515    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512520    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512523    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512526    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512529    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 15:34:39.514770 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512532    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512534    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512537    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512540    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512543    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512546    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512548    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512551    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512553    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512556    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512559    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512561    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512564    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512566    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512569    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512571    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512574    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512576    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512579    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512581    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 15:34:39.515256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512586    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512589    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.512592    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513000    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513007    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513011    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513014    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513018    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513021    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513024    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513026    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513029    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513032    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513035    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513038    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513040    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513043    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513046    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513048    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 15:34:39.515791 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513051    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513054    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513057    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513059    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513062    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513065    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513068    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513070    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513073    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513076    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513078    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513081    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513083    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513087    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513089    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513092    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513095    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513097    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513100    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513102    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 15:34:39.516295 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513105    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513108    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513111    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513113    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513116    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513119    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513121    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513124    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513127    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513130    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513132    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513135    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513137    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513140    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513142    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513145    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513148    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513150    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513153    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513155    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 15:34:39.516782 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513158    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513160    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513163    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513165    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513167    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513170    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513173    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513175    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513178    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513181    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513184    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513186    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513189    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513192    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513195    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513197    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513200    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513203    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513206    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513209    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 15:34:39.517366 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513212    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513215    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513217    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513220    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513222    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513225    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513227    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513230    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513233    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513236    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513239    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513241    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513244    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513247    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513250    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513252    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513254    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513257    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513261    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 15:34:39.517876 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513265    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513268    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513272    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.513275    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513357    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513395    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513409    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513414    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513419    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513423    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513427    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513432    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513435    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513439    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513442    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513446    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513450    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513453    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513456    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513459    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513462    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513464    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513468    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 18 15:34:39.518348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513471    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513475    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513478    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513481    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513485    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513488    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513491    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513495    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513497    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513501    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513504    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513507    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513510    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513515    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513519    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513522    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513524    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513528    2559 flags.go:64] FLAG: --enable-server="true"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513531    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513536    2559 flags.go:64] FLAG: --event-burst="100"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513539    2559 flags.go:64] FLAG: --event-qps="50"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513542    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513546    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513549    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513553    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 15:34:39.518933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513556    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513559    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513562    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513566    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513569    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513572    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513575    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513578    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513581    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513584    2559 flags.go:64] FLAG: --feature-gates=""
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513588    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513592    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513595    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513598    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513602    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513605    2559 flags.go:64] FLAG: --help="false"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513608    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-130-223.ec2.internal"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513611    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513615    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513618    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513621    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513626    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513629    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513632    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 15:34:39.519522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513635    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513638    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513641    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513644    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513647    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513652    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513655    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513658    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513661    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513664    2559 flags.go:64] FLAG: --lock-file=""
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513667    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513670    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513673    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513679    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513681    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513685    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513687    2559 flags.go:64] FLAG: --logging-format="text"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513690    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513693    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513696    2559 flags.go:64] FLAG: --manifest-url=""
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513699    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513703    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513707    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513710    2559 flags.go:64] FLAG: --max-pods="110"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513713    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 15:34:39.520171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513716    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513719    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513722    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513725    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513733    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513736    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513744    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513747    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513751    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513754    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513761    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513767    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513771    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513774    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513777    2559 flags.go:64] FLAG: --port="10250"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513780    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513783    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01dbedb7cb723fc0e"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513786    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513789    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513792    2559 flags.go:64] FLAG: --register-node="true"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513805    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513809    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513812    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513815    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513819    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 18 15:34:39.520818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513823    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513826    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513829    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513832    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513834    2559 flags.go:64] FLAG: --runonce="false"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513837    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513840    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513844    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513847    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513850    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513865    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513868    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513874    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513877    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513880    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513883    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513885    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513888    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513892    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513898    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513904    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513907    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513910    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513914    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513917    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 15:34:39.521450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513920    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513923    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513926    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513929    2559 flags.go:64] FLAG: --v="2"
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513933    2559 flags.go:64] FLAG: --version="false"
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513937    2559 flags.go:64] FLAG: --vmodule=""
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513941    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.513945    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514064    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514068    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514070    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514074    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514078    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514082    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514085    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514088    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514091    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514095    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514098    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514102    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514106    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 15:34:39.522100 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514109    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514112    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514115    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514118    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514121    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514127    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514130    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514133    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514136    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514139    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514142    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514144    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514147    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514150    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514152    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514155    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514158    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514161    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514163    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514166    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 15:34:39.522612 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514169    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514171    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514174    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514177    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514179    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514182    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514185    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514187    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514190    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514193    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514197    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514200    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514202    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514205    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514208    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514210    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514213    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514217    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514220    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514223    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 15:34:39.523137 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514226    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514228    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514231    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514234    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514236    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514239    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514243    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514246    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514249    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514252    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514255    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514258    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514260    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514263    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514266    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514269    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514272    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514275    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514277    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 15:34:39.523622 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514280    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514282    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514285    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514289    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514293    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514296    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514299    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514302    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514305    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514307    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514311    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514314    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514317    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514320    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514322    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514325    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514327    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514330    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514333    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514336    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 15:34:39.524114 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514338    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514341    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514343    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514346    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514349    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514351    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.514354    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.514359    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.521425    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.521449    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.521471    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.521478    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521557    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521563    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521567    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 15:34:39.524635 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521570    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521573    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521576    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521580    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521584    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521588    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521592    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521597    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521601    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521604    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521607    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521610    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521612    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521616    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521619    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521622    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521624    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521627    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521629    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521632    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 15:34:39.525043 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521635    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521637    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521640    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521642    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521645    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521648    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521653    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521657    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521662    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521667    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521672    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521676    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521679    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521682    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521685    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521687    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521690    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521693    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521695    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521698    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 15:34:39.525543 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521701    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521703    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521706    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521709    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521712    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521714    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521717    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521719    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521722    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521725    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521729    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521732    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521735    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521739    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521742    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521746    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521751    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521755    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521759    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 15:34:39.526214 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521762    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521765    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521768    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521770    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521773    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521776    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521779    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521782    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521785    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521788    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521790    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521801    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521804    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521807    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521810    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521813    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521816    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521819    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521823    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521827    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 15:34:39.526706 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521831    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521837    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521840    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521843    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521846    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521849    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521862    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521865    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521867    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521870    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521873    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521876    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521879    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521883    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521886    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521889    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 15:34:39.527226 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.521892    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.521897    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522034    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522040    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522043    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522047    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522050    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522052    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522056    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522063    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522069    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522074    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522077    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522080    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522083    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 15:34:39.527628 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522086    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522089    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522091    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522094    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522097    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522100    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522103    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522105    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522108    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522111    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522114    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522117    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522121    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522124    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522127    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522131    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522134    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522137    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522142    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522146    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 15:34:39.528029 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522151    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522155    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522158    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522161    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522164    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522166    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522169    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522172    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522175    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522178    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522180    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522183    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522186    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522188    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522191    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522194    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522196    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522199    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522201    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 15:34:39.528525 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522204    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522207    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522209    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522212    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522215    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522221    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522225    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522229    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522233    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522237    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522240    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522243    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522246    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522248    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522251    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522253    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522256    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522258    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522261    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 15:34:39.529030 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522263    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522266    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522269    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522271    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522274    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522276    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522279    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522282    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522284    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522287    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522289    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522292    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522296    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522301    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522305    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522309    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522311    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522314    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522317    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522320    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 15:34:39.529494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522322    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 15:34:39.530028 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522325    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 15:34:39.530028 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522327    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 15:34:39.530028 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522331    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 15:34:39.530028 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522334    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 15:34:39.530028 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522336    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 15:34:39.530028 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522339    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 15:34:39.530028 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:39.522341    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 15:34:39.530028 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.522347    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 15:34:39.530028 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.522485    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 15:34:39.530028 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.524644    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 15:34:39.530028 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.525677    2559 server.go:1019] "Starting client certificate rotation"
Sep 18 15:34:39.530028 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.525772    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 15:34:39.530028 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.526735    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 15:34:39.530348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.528960    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 15:34:39.530648 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.530092    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 15:34:39.545415 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.545385    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 18 15:34:39.547237 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.547219    2559 log.go:25] "Validated CRI v1 image API"
Sep 18 15:34:39.548164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.548146    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 15:34:39.550524 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.550495    2559 fs.go:135] Filesystem UUIDs: map[4ec5c30f-7f5f-453f-8766-3a11a931af80:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b3961c80-b991-4d46-b652-13e7f57c0e4e:/dev/nvme0n1p3]
Sep 18 15:34:39.550593 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.550524    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 15:34:39.555771 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.555646    2559 manager.go:216] Machine: {Timestamp:2026-09-18 15:34:39.555150418 +0000 UTC m=+2.116382549 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100221 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec216f0b7371572967a08a118da98c45 SystemUUID:ec216f0b-7371-5729-67a0-8a118da98c45 BootID:2af57871-4709-45a5-b8a3-c559de907b8d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d6:fa:6d:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d6:fa:6d:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:4f:23:de:d1:57 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 15:34:39.555771 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.555765    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 15:34:39.555912 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.555897    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 15:34:39.557570 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.557536    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 15:34:39.557718 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.557574    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-223.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 15:34:39.558838 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.558825    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 15:34:39.558894 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.558840    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 15:34:39.558894 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.558849    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 15:34:39.558894 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.558875    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 15:34:39.558974 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.558953    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 15:34:39.562586 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.562571    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 15:34:39.562718 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.562709    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 15:34:39.562751 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.562745    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 15:34:39.562783 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.562756    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 15:34:39.562783 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.562770    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 15:34:39.562783 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.562779    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 18 15:34:39.562896 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.562788    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 15:34:39.566192 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.566091    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 15:34:39.566192 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.566138    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 15:34:39.568284 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.568251    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 15:34:39.570548 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.570533    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 15:34:39.570614 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.570565    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 15:34:39.571688 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571671    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 15:34:39.571733 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571696    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 15:34:39.571733 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571706    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 15:34:39.571733 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571714    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 15:34:39.571733 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571727    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 15:34:39.571845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571734    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 15:34:39.571845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571753    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 15:34:39.571845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571762    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 15:34:39.571845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571773    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 15:34:39.571845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571780    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 15:34:39.571845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571795    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 15:34:39.571845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571801    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 15:34:39.571845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571826    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 15:34:39.571845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.571831    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 15:34:39.576065 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.576045    2559 server.go:1268] "Started kubelet"
Sep 18 15:34:39.576209 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.576073    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 15:34:39.576209 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.576133    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 15:34:39.576315 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.576252    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 15:34:39.576899 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.576877    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 15:34:39.577218 ip-10-0-130-223 systemd[1]: Started Kubernetes Kubelet.
Sep 18 15:34:39.606123 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.606093    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 15:34:39.606512 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.606484    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 15:34:39.606751 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.606715    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-223.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 15:34:39.606848 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.606684    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 15:34:39.607662 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.607444    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 15:34:39.608802 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.608784    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 15:34:39.613846 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.613818    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 15:34:39.613956 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.613824    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 15:34:39.613956 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.613917    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:39.614265 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.613306    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-223.ec2.internal.18d674368d693208  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-223.ec2.internal,UID:ip-10-0-130-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-223.ec2.internal,},FirstTimestamp:2026-09-18 15:34:39.575888392 +0000 UTC m=+2.137120523,LastTimestamp:2026-09-18 15:34:39.575888392 +0000 UTC m=+2.137120523,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-223.ec2.internal,}"
Sep 18 15:34:39.614464 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.614435    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 15:34:39.614464 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.614456    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 15:34:39.614564 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.614474    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 15:34:39.614609 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.614599    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 15:34:39.614609 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.614606    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 15:34:39.614846 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.614822    2559 factory.go:55] Registering systemd factory
Sep 18 15:34:39.614846 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.614846    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 18 15:34:39.615101 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.615089    2559 factory.go:153] Registering CRI-O factory
Sep 18 15:34:39.615153 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.615103    2559 factory.go:223] Registration of the crio container factory successfully
Sep 18 15:34:39.615226 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.615214    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 15:34:39.615295 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.615234    2559 factory.go:103] Registering Raw factory
Sep 18 15:34:39.615295 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.615246    2559 manager.go:1196] Started watching for new ooms in manager
Sep 18 15:34:39.615696 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.615685    2559 manager.go:349] Starting recovery of all containers
Sep 18 15:34:39.625222 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.625201    2559 manager.go:354] Recovery completed
Sep 18 15:34:39.629603 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.629588    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:39.630058 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.630039    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 15:34:39.630143 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.630124    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 15:34:39.631480 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.631446    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:39.631560 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.631489    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:39.631560 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.631501    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:39.632129 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.632112    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 15:34:39.632129 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.632128    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 15:34:39.632129 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.632136    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:39.632311 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.632147    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 15:34:39.632931 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.632914    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:39.632988 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.632950    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:39.632988 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.632966    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:39.633517 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.633502    2559 policy_none.go:49] "None policy: Start"
Sep 18 15:34:39.633580 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.633521    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 15:34:39.633580 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.633535    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 15:34:39.633642 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.633565    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-223.ec2.internal.18d6743690b966df  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-223.ec2.internal,UID:ip-10-0-130-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-223.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-223.ec2.internal,},FirstTimestamp:2026-09-18 15:34:39.631476447 +0000 UTC m=+2.192708578,LastTimestamp:2026-09-18 15:34:39.631476447 +0000 UTC m=+2.192708578,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-223.ec2.internal,}"
Sep 18 15:34:39.633913 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.633898    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h4fr7"
Sep 18 15:34:39.634796 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.634785    2559 policy_none.go:47] "Start"
Sep 18 15:34:39.652741 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.652708    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h4fr7"
Sep 18 15:34:39.671578 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.671560    2559 manager.go:337] "Starting Device Plugin manager"
Sep 18 15:34:39.677653 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.671599    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 15:34:39.677653 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.671610    2559 server.go:87] "Starting device plugin registration server"
Sep 18 15:34:39.677653 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.671552    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-223.ec2.internal.18d6743690b9b015  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-223.ec2.internal,UID:ip-10-0-130-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-223.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-223.ec2.internal,},FirstTimestamp:2026-09-18 15:34:39.631495189 +0000 UTC m=+2.192727320,LastTimestamp:2026-09-18 15:34:39.631495189 +0000 UTC m=+2.192727320,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-223.ec2.internal,}"
Sep 18 15:34:39.677653 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.671917    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 15:34:39.677653 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.671945    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 15:34:39.677653 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.672165    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 15:34:39.677653 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.672242    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 15:34:39.677653 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.672251    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 15:34:39.677653 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.672734    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 15:34:39.677653 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.672782    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:39.735868 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.735825    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 15:34:39.737027 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.737009    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 15:34:39.737112 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.737077    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 15:34:39.737165 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.737113    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 15:34:39.737217 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.737165    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 15:34:39.747963 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.747940    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 15:34:39.772419 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.772318    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:39.773386 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.773363    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:39.773479 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.773395    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:39.773479 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.773418    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:39.773479 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.773443    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-223.ec2.internal"
Sep 18 15:34:39.801456 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.801428    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-223.ec2.internal"
Sep 18 15:34:39.801548 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.801462    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-223.ec2.internal\": node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:39.821137 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.821107    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:39.837352 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.837321    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal"]
Sep 18 15:34:39.837410 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.837402    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:39.838960 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.838947    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:39.839014 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.838977    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:39.839014 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.838991    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:39.840243 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.840231    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:39.840385 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.840368    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:39.840423 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.840401    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:39.840931 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.840916    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:39.841005 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.840942    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:39.841005 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.840956    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:39.841005 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.840988    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:39.841104 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.841008    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:39.841104 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.841018    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:39.842217 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.842200    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:39.842308 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.842222    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:39.842850 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.842837    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:39.842932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.842874    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:39.842932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.842887    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:39.861589 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.861552    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-223.ec2.internal\" not found" node="ip-10-0-130-223.ec2.internal"
Sep 18 15:34:39.866057 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.866038    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-223.ec2.internal\" not found" node="ip-10-0-130-223.ec2.internal"
Sep 18 15:34:39.915932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.915899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d88c2d4afc35e0e1514f598303c4cf02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal\" (UID: \"d88c2d4afc35e0e1514f598303c4cf02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:39.915932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.915932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d88c2d4afc35e0e1514f598303c4cf02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal\" (UID: \"d88c2d4afc35e0e1514f598303c4cf02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:39.916120 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:39.915950    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b01950708f7973287a6e4968c09a493-config\") pod \"kube-apiserver-proxy-ip-10-0-130-223.ec2.internal\" (UID: \"2b01950708f7973287a6e4968c09a493\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:39.922218 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:39.922197    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:40.016879 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.016823    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d88c2d4afc35e0e1514f598303c4cf02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal\" (UID: \"d88c2d4afc35e0e1514f598303c4cf02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:40.017030 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.016773    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d88c2d4afc35e0e1514f598303c4cf02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal\" (UID: \"d88c2d4afc35e0e1514f598303c4cf02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:40.017030 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.016915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d88c2d4afc35e0e1514f598303c4cf02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal\" (UID: \"d88c2d4afc35e0e1514f598303c4cf02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:40.017030 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.016932    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b01950708f7973287a6e4968c09a493-config\") pod \"kube-apiserver-proxy-ip-10-0-130-223.ec2.internal\" (UID: \"2b01950708f7973287a6e4968c09a493\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:40.017030 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.016965    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b01950708f7973287a6e4968c09a493-config\") pod \"kube-apiserver-proxy-ip-10-0-130-223.ec2.internal\" (UID: \"2b01950708f7973287a6e4968c09a493\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:40.017030 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.016996    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d88c2d4afc35e0e1514f598303c4cf02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal\" (UID: \"d88c2d4afc35e0e1514f598303c4cf02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:40.023037 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:40.022983    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:40.123898 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:40.123830    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:40.164664 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.164631    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:40.169429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.169161    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:40.224128 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:40.224095    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:40.324766 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:40.324679    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:40.425197 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:40.425169    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:40.502463 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.502431    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 15:34:40.525777 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.525746    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 15:34:40.526263 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:40.525748    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:40.526263 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.525931    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 15:34:40.526263 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.525946    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 15:34:40.625915 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:40.625885    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:40.661736 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.661525    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 15:29:39 +0000 UTC" deadline="2028-04-30 09:43:08.136820876 +0000 UTC"
Sep 18 15:34:40.661736 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.661732    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14154h8m27.47509869s"
Sep 18 15:34:40.715478 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:40.715430    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b01950708f7973287a6e4968c09a493.slice/crio-b72270f4f17cfe3f2d5dccc08680ec8be350029e42094b27c5e38f7674f24039 WatchSource:0}: Error finding container b72270f4f17cfe3f2d5dccc08680ec8be350029e42094b27c5e38f7674f24039: Status 404 returned error can't find the container with id b72270f4f17cfe3f2d5dccc08680ec8be350029e42094b27c5e38f7674f24039
Sep 18 15:34:40.715817 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:40.715795    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88c2d4afc35e0e1514f598303c4cf02.slice/crio-5a223ece7939490a63083f649020e0c875a5c1eaa2dd5edab6fd8ca478595201 WatchSource:0}: Error finding container 5a223ece7939490a63083f649020e0c875a5c1eaa2dd5edab6fd8ca478595201: Status 404 returned error can't find the container with id 5a223ece7939490a63083f649020e0c875a5c1eaa2dd5edab6fd8ca478595201
Sep 18 15:34:40.721610 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.721590    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 15:34:40.726378 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:40.726356    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:40.740737 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.740692    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal" event={"ID":"2b01950708f7973287a6e4968c09a493","Type":"ContainerStarted","Data":"b72270f4f17cfe3f2d5dccc08680ec8be350029e42094b27c5e38f7674f24039"}
Sep 18 15:34:40.741743 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.741719    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" event={"ID":"d88c2d4afc35e0e1514f598303c4cf02","Type":"ContainerStarted","Data":"5a223ece7939490a63083f649020e0c875a5c1eaa2dd5edab6fd8ca478595201"}
Sep 18 15:34:40.820320 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.820283    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:40.821504 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.821487    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:40.821573 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.821517    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:40.821573 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:40.821528    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:40.826652 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:40.826631    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:40.927076 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:40.927001    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:41.027662 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:41.027626    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:41.128495 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:41.128461    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-223.ec2.internal\" not found"
Sep 18 15:34:41.187960 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.187931    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 15:34:41.215067 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.215026    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:41.226494 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.226466    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 15:34:41.230972 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.230937    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 15:34:41.231966 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.231944    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal"
Sep 18 15:34:41.250340 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.250238    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 15:34:41.501233 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.501156    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 15:34:41.564205 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.564174    2559 apiserver.go:52] "Watching apiserver"
Sep 18 15:34:41.574941 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.574907    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 15:34:41.575848 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.575826    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn","openshift-cluster-node-tuning-operator/tuned-94hkp","openshift-dns/node-resolver-fpfzs","openshift-image-registry/node-ca-nrqvf","openshift-multus/network-metrics-daemon-7qq7j","openshift-network-diagnostics/network-check-target-5gqw8","kube-system/global-pull-secret-syncer-dzwbg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal","openshift-multus/multus-6bnzx","openshift-multus/multus-additional-cni-plugins-lnfcs","openshift-network-operator/iptables-alerter-th4bc","openshift-ovn-kubernetes/ovnkube-node-6qwct","kube-system/konnectivity-agent-57xz8"]
Sep 18 15:34:41.577554 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.577528    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.581958 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.581938    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 15:34:41.582435 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.582411    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pqbgz\""
Sep 18 15:34:41.582615 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.582596    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 15:34:41.582842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.582827    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 15:34:41.584298 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.584279    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.585869 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.585835    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.586032 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.586006    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 15:34:41.587231 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.587207    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bfldw\""
Sep 18 15:34:41.587501 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.587474    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 15:34:41.587779 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.587763    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fpfzs"
Sep 18 15:34:41.588945 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.588929    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nrqvf"
Sep 18 15:34:41.590255 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.589995    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 15:34:41.590255 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.590073    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4t499\""
Sep 18 15:34:41.590255 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.590133    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 15:34:41.590255 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.590134    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 15:34:41.590490 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.590417    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:41.590544 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:41.590494    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:34:41.591728 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.591710    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-57xz8"
Sep 18 15:34:41.592099 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.592070    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 15:34:41.592246 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.592229    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 15:34:41.592372 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.592355    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 15:34:41.592514 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.592494    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7w6b5\""
Sep 18 15:34:41.592596 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.592563    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dmvm6\""
Sep 18 15:34:41.592914 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.592896    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-th4bc"
Sep 18 15:34:41.593312 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.593292    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 15:34:41.593312 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.593308    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 15:34:41.594496 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.594208    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.595598 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.595575    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:41.595734 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:41.595673    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:34:41.596488 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.596466    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 15:34:41.597164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.596737    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hqh4j\""
Sep 18 15:34:41.597164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.596764    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 15:34:41.597164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.596775    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 15:34:41.597164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.596804    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 15:34:41.597164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.596929    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zq9td\""
Sep 18 15:34:41.597164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.597060    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dhh5k\""
Sep 18 15:34:41.597164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.597077    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 15:34:41.597485 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.597209    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.597485 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.597285    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 15:34:41.597485 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.597460    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 15:34:41.600111 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.600093    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:41.600202 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:41.600150    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:34:41.602785 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.602587    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 15:34:41.602785 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.602655    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 15:34:41.602785 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.602669    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 15:34:41.603011 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.602797    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 15:34:41.603011 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.602823    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tdnv7\""
Sep 18 15:34:41.603011 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.602957    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 15:34:41.603165 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.603061    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 15:34:41.603574 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.603558    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 15:34:41.615352 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.615307    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 15:34:41.615803 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.615752    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 15:34:41.624952 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.624829    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9085c12e-916c-423c-a2da-8ba78ec5fc0c-cnibin\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.624952 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.624888    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xrxc\" (UniqueName: \"kubernetes.io/projected/9085c12e-916c-423c-a2da-8ba78ec5fc0c-kube-api-access-2xrxc\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.624952 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.624917    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/51596704-4c57-46c5-a82a-db2ad011ee51-serviceca\") pod \"node-ca-nrqvf\" (UID: \"51596704-4c57-46c5-a82a-db2ad011ee51\") " pod="openshift-image-registry/node-ca-nrqvf"
Sep 18 15:34:41.624952 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.624941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-var-lib-cni-multus\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.625247 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.624967    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-modprobe-d\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.625247 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.624989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60b93ec7-8e32-4d00-a695-ca96c49a0609-cni-binary-copy\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.625247 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-device-dir\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.625247 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625039    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-sys-fs\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.625247 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625061    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-systemd-units\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.625247 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625089    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-run-openvswitch\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.625247 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625111    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484-iptables-alerter-script\") pod \"iptables-alerter-th4bc\" (UID: \"9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484\") " pod="openshift-network-operator/iptables-alerter-th4bc"
Sep 18 15:34:41.625247 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625239    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac1a1089-9a91-4f0c-a285-79e15b4da235-dbus\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625274    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-kubelet\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625298    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f82eee8-cdc7-4d08-8cee-98827734b5ec-ovnkube-config\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625330    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484-host-slash\") pod \"iptables-alerter-th4bc\" (UID: \"9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484\") " pod="openshift-network-operator/iptables-alerter-th4bc"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625353    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsx7x\" (UniqueName: \"kubernetes.io/projected/9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484-kube-api-access-gsx7x\") pod \"iptables-alerter-th4bc\" (UID: \"9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484\") " pod="openshift-network-operator/iptables-alerter-th4bc"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625382    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-var-lib-cni-bin\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625406    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-var-lib-kubelet\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625428    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-sysctl-conf\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625452    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-slash\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-cni-netd\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625520    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f82eee8-cdc7-4d08-8cee-98827734b5ec-ovn-node-metrics-cert\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625555    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e24a42c3-40e3-4986-bd91-2041896deab6-tmp\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625580    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljflf\" (UniqueName: \"kubernetes.io/projected/792d0ad5-2007-42a0-bb50-a3b4c0ef1af5-kube-api-access-ljflf\") pod \"node-resolver-fpfzs\" (UID: \"792d0ad5-2007-42a0-bb50-a3b4c0ef1af5\") " pod="openshift-dns/node-resolver-fpfzs"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625608    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9085c12e-916c-423c-a2da-8ba78ec5fc0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625635    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntl5s\" (UniqueName: \"kubernetes.io/projected/51596704-4c57-46c5-a82a-db2ad011ee51-kube-api-access-ntl5s\") pod \"node-ca-nrqvf\" (UID: \"51596704-4c57-46c5-a82a-db2ad011ee51\") " pod="openshift-image-registry/node-ca-nrqvf"
Sep 18 15:34:41.625656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625657    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs\") pod \"network-metrics-daemon-7qq7j\" (UID: \"27e17fa7-f7e0-4638-bc66-280734f2d839\") " pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625762    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-multus-socket-dir-parent\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.625967    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f82eee8-cdc7-4d08-8cee-98827734b5ec-ovnkube-script-lib\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626047    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9085c12e-916c-423c-a2da-8ba78ec5fc0c-system-cni-dir\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-cnibin\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626078    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-sysconfig\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-systemd\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-registration-dir\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh5h4\" (UniqueName: \"kubernetes.io/projected/3175f4db-dc38-4c58-8def-81d5f23892dc-kube-api-access-xh5h4\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626192    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-run\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626225    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-lib-modules\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626247    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-host\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626270    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9085c12e-916c-423c-a2da-8ba78ec5fc0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626294    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-hostroot\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626335    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-etc-kubernetes\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626360    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/792d0ad5-2007-42a0-bb50-a3b4c0ef1af5-tmp-dir\") pod \"node-resolver-fpfzs\" (UID: \"792d0ad5-2007-42a0-bb50-a3b4c0ef1af5\") " pod="openshift-dns/node-resolver-fpfzs"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626402    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7e46a8d9-f7b7-469f-99d7-1a13f77741b9-agent-certs\") pod \"konnectivity-agent-57xz8\" (UID: \"7e46a8d9-f7b7-469f-99d7-1a13f77741b9\") " pod="kube-system/konnectivity-agent-57xz8"
Sep 18 15:34:41.626500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626430    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7e46a8d9-f7b7-469f-99d7-1a13f77741b9-konnectivity-ca\") pod \"konnectivity-agent-57xz8\" (UID: \"7e46a8d9-f7b7-469f-99d7-1a13f77741b9\") " pod="kube-system/konnectivity-agent-57xz8"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626455    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626483    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-run-k8s-cni-cncf-io\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626507    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-var-lib-openvswitch\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626530    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51596704-4c57-46c5-a82a-db2ad011ee51-host\") pod \"node-ca-nrqvf\" (UID: \"51596704-4c57-46c5-a82a-db2ad011ee51\") " pod="openshift-image-registry/node-ca-nrqvf"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-run-multus-certs\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626574    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-sysctl-d\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626606    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626628    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9085c12e-916c-423c-a2da-8ba78ec5fc0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626650    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-socket-dir\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626671    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-run-netns\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626694    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-log-socket\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626717    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vv5h\" (UniqueName: \"kubernetes.io/projected/27e17fa7-f7e0-4638-bc66-280734f2d839-kube-api-access-5vv5h\") pod \"network-metrics-daemon-7qq7j\" (UID: \"27e17fa7-f7e0-4638-bc66-280734f2d839\") " pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626740    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-system-cni-dir\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626764    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60b93ec7-8e32-4d00-a695-ca96c49a0609-multus-daemon-config\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrtvt\" (UniqueName: \"kubernetes.io/projected/60b93ec7-8e32-4d00-a695-ca96c49a0609-kube-api-access-qrtvt\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.627335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-etc-openvswitch\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626847    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626887    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac1a1089-9a91-4f0c-a285-79e15b4da235-kubelet-config\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.626924    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-multus-cni-dir\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627218    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-sys\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627298    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh4jh\" (UniqueName: \"kubernetes.io/projected/e24a42c3-40e3-4986-bd91-2041896deab6-kube-api-access-sh4jh\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627322    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-run-systemd\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627343    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-run-ovn\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627376    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-cni-bin\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627398    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w58pr\" (UniqueName: \"kubernetes.io/projected/4f82eee8-cdc7-4d08-8cee-98827734b5ec-kube-api-access-w58pr\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627412    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-kubernetes\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e24a42c3-40e3-4986-bd91-2041896deab6-etc-tuned\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627494    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f82eee8-cdc7-4d08-8cee-98827734b5ec-env-overrides\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627545    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9085c12e-916c-423c-a2da-8ba78ec5fc0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627579    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-run-netns\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.628172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627615    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-multus-conf-dir\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.628875 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627639    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-var-lib-kubelet\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.628875 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627661    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/792d0ad5-2007-42a0-bb50-a3b4c0ef1af5-hosts-file\") pod \"node-resolver-fpfzs\" (UID: \"792d0ad5-2007-42a0-bb50-a3b4c0ef1af5\") " pod="openshift-dns/node-resolver-fpfzs"
Sep 18 15:34:41.628875 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627684    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.628875 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dps2\" (UniqueName: \"kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2\") pod \"network-check-target-5gqw8\" (UID: \"6b68ac73-f537-4ac6-af11-6accaf8431f9\") " pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:41.628875 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.627924    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-node-log\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.630389 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.630364    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 15:34:41.677913 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.677881    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7tf8z"
Sep 18 15:34:41.686342 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.686310    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7tf8z"
Sep 18 15:34:41.729116 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9085c12e-916c-423c-a2da-8ba78ec5fc0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.729116 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntl5s\" (UniqueName: \"kubernetes.io/projected/51596704-4c57-46c5-a82a-db2ad011ee51-kube-api-access-ntl5s\") pod \"node-ca-nrqvf\" (UID: \"51596704-4c57-46c5-a82a-db2ad011ee51\") " pod="openshift-image-registry/node-ca-nrqvf"
Sep 18 15:34:41.729356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs\") pod \"network-metrics-daemon-7qq7j\" (UID: \"27e17fa7-f7e0-4638-bc66-280734f2d839\") " pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:41.729356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-multus-socket-dir-parent\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.729356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729170    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f82eee8-cdc7-4d08-8cee-98827734b5ec-ovnkube-script-lib\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.729356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729194    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9085c12e-916c-423c-a2da-8ba78ec5fc0c-system-cni-dir\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.729356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-cnibin\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.729356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-sysconfig\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.729356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729269    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-systemd\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.729356 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:41.729279    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:41.729356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729337    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-registration-dir\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:41.729384    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs podName:27e17fa7-f7e0-4638-bc66-280734f2d839 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:42.229339153 +0000 UTC m=+4.790571272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs") pod "network-metrics-daemon-7qq7j" (UID: "27e17fa7-f7e0-4638-bc66-280734f2d839") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729290    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-registration-dir\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh5h4\" (UniqueName: \"kubernetes.io/projected/3175f4db-dc38-4c58-8def-81d5f23892dc-kube-api-access-xh5h4\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729454    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-run\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729479    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-lib-modules\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-multus-socket-dir-parent\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729549    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-host\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729596    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-sysconfig\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729613    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-run\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729634    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-systemd\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729502    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-host\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9085c12e-916c-423c-a2da-8ba78ec5fc0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729675    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-cnibin\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729688    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-hostroot\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.729725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729723    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-hostroot\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729771    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9085c12e-916c-423c-a2da-8ba78ec5fc0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-lib-modules\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729836    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-etc-kubernetes\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729845    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9085c12e-916c-423c-a2da-8ba78ec5fc0c-system-cni-dir\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729884    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/792d0ad5-2007-42a0-bb50-a3b4c0ef1af5-tmp-dir\") pod \"node-resolver-fpfzs\" (UID: \"792d0ad5-2007-42a0-bb50-a3b4c0ef1af5\") " pod="openshift-dns/node-resolver-fpfzs"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729902    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-etc-kubernetes\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.729909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7e46a8d9-f7b7-469f-99d7-1a13f77741b9-agent-certs\") pod \"konnectivity-agent-57xz8\" (UID: \"7e46a8d9-f7b7-469f-99d7-1a13f77741b9\") " pod="kube-system/konnectivity-agent-57xz8"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9085c12e-916c-423c-a2da-8ba78ec5fc0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730177    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/792d0ad5-2007-42a0-bb50-a3b4c0ef1af5-tmp-dir\") pod \"node-resolver-fpfzs\" (UID: \"792d0ad5-2007-42a0-bb50-a3b4c0ef1af5\") " pod="openshift-dns/node-resolver-fpfzs"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730197    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7e46a8d9-f7b7-469f-99d7-1a13f77741b9-konnectivity-ca\") pod \"konnectivity-agent-57xz8\" (UID: \"7e46a8d9-f7b7-469f-99d7-1a13f77741b9\") " pod="kube-system/konnectivity-agent-57xz8"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:41.730294    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730296    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f82eee8-cdc7-4d08-8cee-98827734b5ec-ovnkube-script-lib\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-run-k8s-cni-cncf-io\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.730356 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:41.730355    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret podName:ac1a1089-9a91-4f0c-a285-79e15b4da235 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:42.230339162 +0000 UTC m=+4.791571289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret") pod "global-pull-secret-syncer-dzwbg" (UID: "ac1a1089-9a91-4f0c-a285-79e15b4da235") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-var-lib-openvswitch\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730385    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-run-k8s-cni-cncf-io\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51596704-4c57-46c5-a82a-db2ad011ee51-host\") pod \"node-ca-nrqvf\" (UID: \"51596704-4c57-46c5-a82a-db2ad011ee51\") " pod="openshift-image-registry/node-ca-nrqvf"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-var-lib-openvswitch\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730433    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-run-multus-certs\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730455    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51596704-4c57-46c5-a82a-db2ad011ee51-host\") pod \"node-ca-nrqvf\" (UID: \"51596704-4c57-46c5-a82a-db2ad011ee51\") " pod="openshift-image-registry/node-ca-nrqvf"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-sysctl-d\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730501    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9085c12e-916c-423c-a2da-8ba78ec5fc0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730522    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-socket-dir\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-run-netns\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730582    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-log-socket\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730604    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-run-multus-certs\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vv5h\" (UniqueName: \"kubernetes.io/projected/27e17fa7-f7e0-4638-bc66-280734f2d839-kube-api-access-5vv5h\") pod \"network-metrics-daemon-7qq7j\" (UID: \"27e17fa7-f7e0-4638-bc66-280734f2d839\") " pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730644    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7e46a8d9-f7b7-469f-99d7-1a13f77741b9-konnectivity-ca\") pod \"konnectivity-agent-57xz8\" (UID: \"7e46a8d9-f7b7-469f-99d7-1a13f77741b9\") " pod="kube-system/konnectivity-agent-57xz8"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-system-cni-dir\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.730997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60b93ec7-8e32-4d00-a695-ca96c49a0609-multus-daemon-config\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730696    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-socket-dir\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730559    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-sysctl-d\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrtvt\" (UniqueName: \"kubernetes.io/projected/60b93ec7-8e32-4d00-a695-ca96c49a0609-kube-api-access-qrtvt\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730743    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-log-socket\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-etc-openvswitch\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730799    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-run-netns\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730822    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9085c12e-916c-423c-a2da-8ba78ec5fc0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac1a1089-9a91-4f0c-a285-79e15b4da235-kubelet-config\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730918    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-multus-cni-dir\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730943    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-sys\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730947    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-etc-openvswitch\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-system-cni-dir\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730951    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.730982    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh4jh\" (UniqueName: \"kubernetes.io/projected/e24a42c3-40e3-4986-bd91-2041896deab6-kube-api-access-sh4jh\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.731842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731008    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-run-systemd\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731048    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-sys\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731075    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-multus-cni-dir\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731097    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-run-ovn\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731114    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac1a1089-9a91-4f0c-a285-79e15b4da235-kubelet-config\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731113    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-run-systemd\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731135    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-cni-bin\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731141    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-run-ovn\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731172    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w58pr\" (UniqueName: \"kubernetes.io/projected/4f82eee8-cdc7-4d08-8cee-98827734b5ec-kube-api-access-w58pr\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731173    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60b93ec7-8e32-4d00-a695-ca96c49a0609-multus-daemon-config\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731219    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-kubernetes\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731225    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-cni-bin\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731302    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e24a42c3-40e3-4986-bd91-2041896deab6-etc-tuned\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731331    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f82eee8-cdc7-4d08-8cee-98827734b5ec-env-overrides\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731337    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731345    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-kubernetes\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9085c12e-916c-423c-a2da-8ba78ec5fc0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.732645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731387    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-run-netns\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731401    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-multus-conf-dir\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-var-lib-kubelet\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731456    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-run-netns\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731468    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-var-lib-kubelet\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731491    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-multus-conf-dir\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731494    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/792d0ad5-2007-42a0-bb50-a3b4c0ef1af5-hosts-file\") pod \"node-resolver-fpfzs\" (UID: \"792d0ad5-2007-42a0-bb50-a3b4c0ef1af5\") " pod="openshift-dns/node-resolver-fpfzs"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731519    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dps2\" (UniqueName: \"kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2\") pod \"network-check-target-5gqw8\" (UID: \"6b68ac73-f537-4ac6-af11-6accaf8431f9\") " pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731577    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-node-log\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9085c12e-916c-423c-a2da-8ba78ec5fc0c-cnibin\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731613    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/792d0ad5-2007-42a0-bb50-a3b4c0ef1af5-hosts-file\") pod \"node-resolver-fpfzs\" (UID: \"792d0ad5-2007-42a0-bb50-a3b4c0ef1af5\") " pod="openshift-dns/node-resolver-fpfzs"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xrxc\" (UniqueName: \"kubernetes.io/projected/9085c12e-916c-423c-a2da-8ba78ec5fc0c-kube-api-access-2xrxc\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731676    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-node-log\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731677    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/51596704-4c57-46c5-a82a-db2ad011ee51-serviceca\") pod \"node-ca-nrqvf\" (UID: \"51596704-4c57-46c5-a82a-db2ad011ee51\") " pod="openshift-image-registry/node-ca-nrqvf"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-var-lib-cni-multus\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f82eee8-cdc7-4d08-8cee-98827734b5ec-env-overrides\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.733398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731737    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-modprobe-d\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731761    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60b93ec7-8e32-4d00-a695-ca96c49a0609-cni-binary-copy\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731785    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-device-dir\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731788    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-var-lib-cni-multus\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-sys-fs\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731836    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-systemd-units\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731840    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9085c12e-916c-423c-a2da-8ba78ec5fc0c-cnibin\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-run-openvswitch\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731903    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484-iptables-alerter-script\") pod \"iptables-alerter-th4bc\" (UID: \"9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484\") " pod="openshift-network-operator/iptables-alerter-th4bc"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731909    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9085c12e-916c-423c-a2da-8ba78ec5fc0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731964    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-run-openvswitch\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731966    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-device-dir\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731904    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-modprobe-d\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.731987    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac1a1089-9a91-4f0c-a285-79e15b4da235-dbus\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732036    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3175f4db-dc38-4c58-8def-81d5f23892dc-sys-fs\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-kubelet\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732094    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f82eee8-cdc7-4d08-8cee-98827734b5ec-ovnkube-config\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.734105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732072    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/51596704-4c57-46c5-a82a-db2ad011ee51-serviceca\") pod \"node-ca-nrqvf\" (UID: \"51596704-4c57-46c5-a82a-db2ad011ee51\") " pod="openshift-image-registry/node-ca-nrqvf"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732126    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-systemd-units\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732129    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac1a1089-9a91-4f0c-a285-79e15b4da235-dbus\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732186    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484-host-slash\") pod \"iptables-alerter-th4bc\" (UID: \"9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484\") " pod="openshift-network-operator/iptables-alerter-th4bc"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732215    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsx7x\" (UniqueName: \"kubernetes.io/projected/9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484-kube-api-access-gsx7x\") pod \"iptables-alerter-th4bc\" (UID: \"9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484\") " pod="openshift-network-operator/iptables-alerter-th4bc"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732238    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-var-lib-cni-bin\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-kubelet\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732264    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-var-lib-kubelet\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732289    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-sysctl-conf\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732311    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60b93ec7-8e32-4d00-a695-ca96c49a0609-cni-binary-copy\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-var-lib-cni-bin\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732385    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-slash\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732312    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-slash\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732409    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60b93ec7-8e32-4d00-a695-ca96c49a0609-host-var-lib-kubelet\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732428    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-cni-netd\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732454    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f82eee8-cdc7-4d08-8cee-98827734b5ec-ovn-node-metrics-cert\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732477    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e24a42c3-40e3-4986-bd91-2041896deab6-tmp\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732481    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484-host-slash\") pod \"iptables-alerter-th4bc\" (UID: \"9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484\") " pod="openshift-network-operator/iptables-alerter-th4bc"
Sep 18 15:34:41.735061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732491    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484-iptables-alerter-script\") pod \"iptables-alerter-th4bc\" (UID: \"9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484\") " pod="openshift-network-operator/iptables-alerter-th4bc"
Sep 18 15:34:41.735986 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732502    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljflf\" (UniqueName: \"kubernetes.io/projected/792d0ad5-2007-42a0-bb50-a3b4c0ef1af5-kube-api-access-ljflf\") pod \"node-resolver-fpfzs\" (UID: \"792d0ad5-2007-42a0-bb50-a3b4c0ef1af5\") " pod="openshift-dns/node-resolver-fpfzs"
Sep 18 15:34:41.735986 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732516    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e24a42c3-40e3-4986-bd91-2041896deab6-etc-sysctl-conf\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.735986 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732555    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f82eee8-cdc7-4d08-8cee-98827734b5ec-host-cni-netd\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.735986 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.732580    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f82eee8-cdc7-4d08-8cee-98827734b5ec-ovnkube-config\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.735986 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.734975    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e24a42c3-40e3-4986-bd91-2041896deab6-tmp\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.735986 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.735036    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e24a42c3-40e3-4986-bd91-2041896deab6-etc-tuned\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.735986 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.735076    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f82eee8-cdc7-4d08-8cee-98827734b5ec-ovn-node-metrics-cert\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.735986 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.735168    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7e46a8d9-f7b7-469f-99d7-1a13f77741b9-agent-certs\") pod \"konnectivity-agent-57xz8\" (UID: \"7e46a8d9-f7b7-469f-99d7-1a13f77741b9\") " pod="kube-system/konnectivity-agent-57xz8"
Sep 18 15:34:41.755166 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.755092    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w58pr\" (UniqueName: \"kubernetes.io/projected/4f82eee8-cdc7-4d08-8cee-98827734b5ec-kube-api-access-w58pr\") pod \"ovnkube-node-6qwct\" (UID: \"4f82eee8-cdc7-4d08-8cee-98827734b5ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:41.756698 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:41.756677    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:34:41.756838 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:41.756707    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:34:41.756838 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:41.756721    2559 projected.go:196] Error preparing data for projected volume kube-api-access-4dps2 for pod openshift-network-diagnostics/network-check-target-5gqw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:41.756838 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:41.756780    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2 podName:6b68ac73-f537-4ac6-af11-6accaf8431f9 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:42.256765787 +0000 UTC m=+4.817997915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4dps2" (UniqueName: "kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2") pod "network-check-target-5gqw8" (UID: "6b68ac73-f537-4ac6-af11-6accaf8431f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:41.757959 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.757935    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljflf\" (UniqueName: \"kubernetes.io/projected/792d0ad5-2007-42a0-bb50-a3b4c0ef1af5-kube-api-access-ljflf\") pod \"node-resolver-fpfzs\" (UID: \"792d0ad5-2007-42a0-bb50-a3b4c0ef1af5\") " pod="openshift-dns/node-resolver-fpfzs"
Sep 18 15:34:41.759036 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.759016    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh4jh\" (UniqueName: \"kubernetes.io/projected/e24a42c3-40e3-4986-bd91-2041896deab6-kube-api-access-sh4jh\") pod \"tuned-94hkp\" (UID: \"e24a42c3-40e3-4986-bd91-2041896deab6\") " pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.761455 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.761423    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsx7x\" (UniqueName: \"kubernetes.io/projected/9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484-kube-api-access-gsx7x\") pod \"iptables-alerter-th4bc\" (UID: \"9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484\") " pod="openshift-network-operator/iptables-alerter-th4bc"
Sep 18 15:34:41.761571 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.761463    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh5h4\" (UniqueName: \"kubernetes.io/projected/3175f4db-dc38-4c58-8def-81d5f23892dc-kube-api-access-xh5h4\") pod \"aws-ebs-csi-driver-node-nplgn\" (UID: \"3175f4db-dc38-4c58-8def-81d5f23892dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.761974 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.761948    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntl5s\" (UniqueName: \"kubernetes.io/projected/51596704-4c57-46c5-a82a-db2ad011ee51-kube-api-access-ntl5s\") pod \"node-ca-nrqvf\" (UID: \"51596704-4c57-46c5-a82a-db2ad011ee51\") " pod="openshift-image-registry/node-ca-nrqvf"
Sep 18 15:34:41.762248 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.762229    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xrxc\" (UniqueName: \"kubernetes.io/projected/9085c12e-916c-423c-a2da-8ba78ec5fc0c-kube-api-access-2xrxc\") pod \"multus-additional-cni-plugins-lnfcs\" (UID: \"9085c12e-916c-423c-a2da-8ba78ec5fc0c\") " pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.762817 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.762796    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vv5h\" (UniqueName: \"kubernetes.io/projected/27e17fa7-f7e0-4638-bc66-280734f2d839-kube-api-access-5vv5h\") pod \"network-metrics-daemon-7qq7j\" (UID: \"27e17fa7-f7e0-4638-bc66-280734f2d839\") " pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:41.763175 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.763151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrtvt\" (UniqueName: \"kubernetes.io/projected/60b93ec7-8e32-4d00-a695-ca96c49a0609-kube-api-access-qrtvt\") pod \"multus-6bnzx\" (UID: \"60b93ec7-8e32-4d00-a695-ca96c49a0609\") " pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.895038 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.894994    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6bnzx"
Sep 18 15:34:41.903703 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.903667    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lnfcs"
Sep 18 15:34:41.913752 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.913722    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-94hkp"
Sep 18 15:34:41.920959 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.920933    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fpfzs"
Sep 18 15:34:41.930840 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.930814    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nrqvf"
Sep 18 15:34:41.939335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.939313    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-57xz8"
Sep 18 15:34:41.949020 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.948993    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-th4bc"
Sep 18 15:34:41.957812 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.957779    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn"
Sep 18 15:34:41.964746 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:41.964719    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:34:42.236194 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.236160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs\") pod \"network-metrics-daemon-7qq7j\" (UID: \"27e17fa7-f7e0-4638-bc66-280734f2d839\") " pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:42.236382 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.236201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:42.236382 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:42.236308    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:42.236382 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:42.236314    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:42.236382 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:42.236362    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret podName:ac1a1089-9a91-4f0c-a285-79e15b4da235 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:43.236347407 +0000 UTC m=+5.797579527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret") pod "global-pull-secret-syncer-dzwbg" (UID: "ac1a1089-9a91-4f0c-a285-79e15b4da235") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:42.236382 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:42.236376    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs podName:27e17fa7-f7e0-4638-bc66-280734f2d839 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:43.236369883 +0000 UTC m=+5.797602002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs") pod "network-metrics-daemon-7qq7j" (UID: "27e17fa7-f7e0-4638-bc66-280734f2d839") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:42.337411 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.337372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dps2\" (UniqueName: \"kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2\") pod \"network-check-target-5gqw8\" (UID: \"6b68ac73-f537-4ac6-af11-6accaf8431f9\") " pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:42.337575 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:42.337558    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:34:42.337650 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:42.337585    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:34:42.337650 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:42.337599    2559 projected.go:196] Error preparing data for projected volume kube-api-access-4dps2 for pod openshift-network-diagnostics/network-check-target-5gqw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:42.337798 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:42.337667    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2 podName:6b68ac73-f537-4ac6-af11-6accaf8431f9 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:43.337647099 +0000 UTC m=+5.898879232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dps2" (UniqueName: "kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2") pod "network-check-target-5gqw8" (UID: "6b68ac73-f537-4ac6-af11-6accaf8431f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:42.580609 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:42.580578    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9085c12e_916c_423c_a2da_8ba78ec5fc0c.slice/crio-8fa49bbf290e505013e7486405e6a33cd95ada8af774070377f11e2ea6bd0f3f WatchSource:0}: Error finding container 8fa49bbf290e505013e7486405e6a33cd95ada8af774070377f11e2ea6bd0f3f: Status 404 returned error can't find the container with id 8fa49bbf290e505013e7486405e6a33cd95ada8af774070377f11e2ea6bd0f3f
Sep 18 15:34:42.587175 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:42.587143    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792d0ad5_2007_42a0_bb50_a3b4c0ef1af5.slice/crio-73ccfc5f93d72f90882d39a84c567dc63d7f19309850195c66887d4d7cabffe3 WatchSource:0}: Error finding container 73ccfc5f93d72f90882d39a84c567dc63d7f19309850195c66887d4d7cabffe3: Status 404 returned error can't find the container with id 73ccfc5f93d72f90882d39a84c567dc63d7f19309850195c66887d4d7cabffe3
Sep 18 15:34:42.606256 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:42.606226    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b93ec7_8e32_4d00_a695_ca96c49a0609.slice/crio-57b0bbeb1a209069036bf7a4728a6476f94c6bf89f1365bc14a64071b4901888 WatchSource:0}: Error finding container 57b0bbeb1a209069036bf7a4728a6476f94c6bf89f1365bc14a64071b4901888: Status 404 returned error can't find the container with id 57b0bbeb1a209069036bf7a4728a6476f94c6bf89f1365bc14a64071b4901888
Sep 18 15:34:42.607990 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:42.607944    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24a42c3_40e3_4986_bd91_2041896deab6.slice/crio-31df152ad499c157a2d23d8d7dfec4082f3d8f94e09675c3a370c9b8436bc0ad WatchSource:0}: Error finding container 31df152ad499c157a2d23d8d7dfec4082f3d8f94e09675c3a370c9b8436bc0ad: Status 404 returned error can't find the container with id 31df152ad499c157a2d23d8d7dfec4082f3d8f94e09675c3a370c9b8436bc0ad
Sep 18 15:34:42.610404 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:42.610377    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3175f4db_dc38_4c58_8def_81d5f23892dc.slice/crio-59a0a63d38187b71fe9e5837203ff8b6a250449432aa84f9eebae62c9f11a4c2 WatchSource:0}: Error finding container 59a0a63d38187b71fe9e5837203ff8b6a250449432aa84f9eebae62c9f11a4c2: Status 404 returned error can't find the container with id 59a0a63d38187b71fe9e5837203ff8b6a250449432aa84f9eebae62c9f11a4c2
Sep 18 15:34:42.611268 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:42.611246    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e46a8d9_f7b7_469f_99d7_1a13f77741b9.slice/crio-29d4cb0f01574da9489507cbeb3ca25ea10dcc4187710b077fe9f4d446850b93 WatchSource:0}: Error finding container 29d4cb0f01574da9489507cbeb3ca25ea10dcc4187710b077fe9f4d446850b93: Status 404 returned error can't find the container with id 29d4cb0f01574da9489507cbeb3ca25ea10dcc4187710b077fe9f4d446850b93
Sep 18 15:34:42.612490 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:34:42.612473    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51596704_4c57_46c5_a82a_db2ad011ee51.slice/crio-51436874cf231215306a2e6045d0884d7c4f1b0da618779556f58b10469bc4c7 WatchSource:0}: Error finding container 51436874cf231215306a2e6045d0884d7c4f1b0da618779556f58b10469bc4c7: Status 404 returned error can't find the container with id 51436874cf231215306a2e6045d0884d7c4f1b0da618779556f58b10469bc4c7
Sep 18 15:34:42.689481 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.689446    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 15:29:41 +0000 UTC" deadline="2028-04-23 12:32:07.846697416 +0000 UTC"
Sep 18 15:34:42.689481 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.689477    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13988h57m25.157224361s"
Sep 18 15:34:42.748604 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.748571    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-th4bc" event={"ID":"9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484","Type":"ContainerStarted","Data":"3cd502c5ca324206a66abbc0ccb860dce2d750d4b979bb84b5148ba3017b0e55"}
Sep 18 15:34:42.751070 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.751037    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct" event={"ID":"4f82eee8-cdc7-4d08-8cee-98827734b5ec","Type":"ContainerStarted","Data":"db7d9981fc7fe732fc831223210e3f6acf07397df8959e598c57685d6e9ee186"}
Sep 18 15:34:42.752259 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.752234    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nrqvf" event={"ID":"51596704-4c57-46c5-a82a-db2ad011ee51","Type":"ContainerStarted","Data":"51436874cf231215306a2e6045d0884d7c4f1b0da618779556f58b10469bc4c7"}
Sep 18 15:34:42.753347 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.753328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn" event={"ID":"3175f4db-dc38-4c58-8def-81d5f23892dc","Type":"ContainerStarted","Data":"59a0a63d38187b71fe9e5837203ff8b6a250449432aa84f9eebae62c9f11a4c2"}
Sep 18 15:34:42.755794 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.755764    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-94hkp" event={"ID":"e24a42c3-40e3-4986-bd91-2041896deab6","Type":"ContainerStarted","Data":"31df152ad499c157a2d23d8d7dfec4082f3d8f94e09675c3a370c9b8436bc0ad"}
Sep 18 15:34:42.756867 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.756834    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6bnzx" event={"ID":"60b93ec7-8e32-4d00-a695-ca96c49a0609","Type":"ContainerStarted","Data":"57b0bbeb1a209069036bf7a4728a6476f94c6bf89f1365bc14a64071b4901888"}
Sep 18 15:34:42.759424 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.759400    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fpfzs" event={"ID":"792d0ad5-2007-42a0-bb50-a3b4c0ef1af5","Type":"ContainerStarted","Data":"73ccfc5f93d72f90882d39a84c567dc63d7f19309850195c66887d4d7cabffe3"}
Sep 18 15:34:42.761303 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.761283    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal" event={"ID":"2b01950708f7973287a6e4968c09a493","Type":"ContainerStarted","Data":"26bf7749571ff5f158ead6712d817fce7375e2c68611531460ce94b5a2a9dacd"}
Sep 18 15:34:42.762465 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.762438    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-57xz8" event={"ID":"7e46a8d9-f7b7-469f-99d7-1a13f77741b9","Type":"ContainerStarted","Data":"29d4cb0f01574da9489507cbeb3ca25ea10dcc4187710b077fe9f4d446850b93"}
Sep 18 15:34:42.763564 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:42.763545    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnfcs" event={"ID":"9085c12e-916c-423c-a2da-8ba78ec5fc0c","Type":"ContainerStarted","Data":"8fa49bbf290e505013e7486405e6a33cd95ada8af774070377f11e2ea6bd0f3f"}
Sep 18 15:34:43.176089 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:43.175479    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 15:34:43.245177 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:43.244405    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs\") pod \"network-metrics-daemon-7qq7j\" (UID: \"27e17fa7-f7e0-4638-bc66-280734f2d839\") " pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:43.245177 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:43.244461    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:43.245177 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:43.244598    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:43.245177 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:43.244660    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret podName:ac1a1089-9a91-4f0c-a285-79e15b4da235 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:45.244641089 +0000 UTC m=+7.805873221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret") pod "global-pull-secret-syncer-dzwbg" (UID: "ac1a1089-9a91-4f0c-a285-79e15b4da235") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:43.245177 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:43.245068    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:43.245177 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:43.245117    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs podName:27e17fa7-f7e0-4638-bc66-280734f2d839 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:45.245102116 +0000 UTC m=+7.806334247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs") pod "network-metrics-daemon-7qq7j" (UID: "27e17fa7-f7e0-4638-bc66-280734f2d839") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:43.345956 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:43.345330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dps2\" (UniqueName: \"kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2\") pod \"network-check-target-5gqw8\" (UID: \"6b68ac73-f537-4ac6-af11-6accaf8431f9\") " pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:43.345956 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:43.345499    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:34:43.345956 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:43.345521    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:34:43.345956 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:43.345534    2559 projected.go:196] Error preparing data for projected volume kube-api-access-4dps2 for pod openshift-network-diagnostics/network-check-target-5gqw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:43.345956 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:43.345594    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2 podName:6b68ac73-f537-4ac6-af11-6accaf8431f9 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:45.345575942 +0000 UTC m=+7.906808076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dps2" (UniqueName: "kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2") pod "network-check-target-5gqw8" (UID: "6b68ac73-f537-4ac6-af11-6accaf8431f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:43.690606 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:43.690532    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 15:29:41 +0000 UTC" deadline="2028-02-14 18:59:30.908914205 +0000 UTC"
Sep 18 15:34:43.690606 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:43.690605    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12339h24m47.218314085s"
Sep 18 15:34:43.738154 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:43.737622    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:43.738154 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:43.737622    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:43.738154 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:43.737776    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:34:43.738154 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:43.737812    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:34:43.738154 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:43.737627    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:43.738154 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:43.737941    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:34:43.777836 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:43.777790    2559 generic.go:356] "Generic (PLEG): container finished" podID="d88c2d4afc35e0e1514f598303c4cf02" containerID="a1172a8e9c1440564754eec05808e2da7ce5d02217f7a3f2cc5369fecbcafc19" exitCode=0
Sep 18 15:34:43.778772 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:43.778743    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" event={"ID":"d88c2d4afc35e0e1514f598303c4cf02","Type":"ContainerDied","Data":"a1172a8e9c1440564754eec05808e2da7ce5d02217f7a3f2cc5369fecbcafc19"}
Sep 18 15:34:43.809452 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:43.809159    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-223.ec2.internal" podStartSLOduration=2.8091444020000003 podStartE2EDuration="2.809144402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:42.795521119 +0000 UTC m=+5.356753258" watchObservedRunningTime="2026-09-18 15:34:43.809144402 +0000 UTC m=+6.370376546"
Sep 18 15:34:44.806141 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:44.806102    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" event={"ID":"d88c2d4afc35e0e1514f598303c4cf02","Type":"ContainerStarted","Data":"41f66cdedaee46d2639e0445da04c36c967c365a649b493a8d082cf81063613f"}
Sep 18 15:34:45.263516 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:45.263483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs\") pod \"network-metrics-daemon-7qq7j\" (UID: \"27e17fa7-f7e0-4638-bc66-280734f2d839\") " pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:45.263699 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:45.263534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:45.263699 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:45.263643    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:45.263821 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:45.263726    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret podName:ac1a1089-9a91-4f0c-a285-79e15b4da235 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:49.263712755 +0000 UTC m=+11.824944872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret") pod "global-pull-secret-syncer-dzwbg" (UID: "ac1a1089-9a91-4f0c-a285-79e15b4da235") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:45.264215 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:45.264092    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:45.264215 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:45.264155    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs podName:27e17fa7-f7e0-4638-bc66-280734f2d839 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:49.264138774 +0000 UTC m=+11.825370893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs") pod "network-metrics-daemon-7qq7j" (UID: "27e17fa7-f7e0-4638-bc66-280734f2d839") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:45.364585 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:45.363886    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dps2\" (UniqueName: \"kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2\") pod \"network-check-target-5gqw8\" (UID: \"6b68ac73-f537-4ac6-af11-6accaf8431f9\") " pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:45.364585 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:45.364107    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:34:45.364585 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:45.364128    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:34:45.364585 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:45.364141    2559 projected.go:196] Error preparing data for projected volume kube-api-access-4dps2 for pod openshift-network-diagnostics/network-check-target-5gqw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:45.364585 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:45.364206    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2 podName:6b68ac73-f537-4ac6-af11-6accaf8431f9 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:49.364184527 +0000 UTC m=+11.925416650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dps2" (UniqueName: "kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2") pod "network-check-target-5gqw8" (UID: "6b68ac73-f537-4ac6-af11-6accaf8431f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:45.738156 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:45.738005    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:45.738156 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:45.738149    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:34:45.739025 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:45.738608    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:45.739025 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:45.738709    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:34:45.739025 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:45.738833    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:45.739025 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:45.738947    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:34:47.738026 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:47.737634    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:47.738026 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:47.737768    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:34:47.738026 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:47.737639    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:47.738026 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:47.737901    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:34:47.738026 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:47.737925    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:47.738026 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:47.737991    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:34:49.296017 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:49.295974    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs\") pod \"network-metrics-daemon-7qq7j\" (UID: \"27e17fa7-f7e0-4638-bc66-280734f2d839\") " pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:49.296586 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:49.296033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:49.296586 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:49.296143    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:49.296586 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:49.296223    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs podName:27e17fa7-f7e0-4638-bc66-280734f2d839 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:57.296200696 +0000 UTC m=+19.857432831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs") pod "network-metrics-daemon-7qq7j" (UID: "27e17fa7-f7e0-4638-bc66-280734f2d839") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:49.296586 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:49.296152    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:49.296586 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:49.296304    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret podName:ac1a1089-9a91-4f0c-a285-79e15b4da235 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:57.296284977 +0000 UTC m=+19.857517108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret") pod "global-pull-secret-syncer-dzwbg" (UID: "ac1a1089-9a91-4f0c-a285-79e15b4da235") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:49.396946 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:49.396913    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dps2\" (UniqueName: \"kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2\") pod \"network-check-target-5gqw8\" (UID: \"6b68ac73-f537-4ac6-af11-6accaf8431f9\") " pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:49.397126 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:49.397102    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:34:49.397181 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:49.397134    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:34:49.397181 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:49.397149    2559 projected.go:196] Error preparing data for projected volume kube-api-access-4dps2 for pod openshift-network-diagnostics/network-check-target-5gqw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:49.397255 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:49.397217    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2 podName:6b68ac73-f537-4ac6-af11-6accaf8431f9 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:57.397195312 +0000 UTC m=+19.958427476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dps2" (UniqueName: "kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2") pod "network-check-target-5gqw8" (UID: "6b68ac73-f537-4ac6-af11-6accaf8431f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:49.738662 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:49.738163    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:49.738662 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:49.738279    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:34:49.738662 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:49.738310    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:49.738662 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:49.738391    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:34:49.738662 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:49.738429    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:49.738662 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:49.738502    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:34:51.737972 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:51.737939    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:51.738849 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:51.738386    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:51.738849 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:51.738394    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:34:51.741681 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:51.741659    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:51.741681 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:51.741660    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:34:51.742281 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:51.742239    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:34:53.738025 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:53.737988    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:53.738466 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:53.738000    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:53.738466 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:53.738128    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:34:53.738466 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:53.738007    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:53.738466 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:53.738199    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:34:53.738466 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:53.738251    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:34:55.737842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:55.737807    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:55.737842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:55.737843    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:55.738368 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:55.737807    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:55.738368 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:55.737977    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:34:55.738368 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:55.738086    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:34:55.738368 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:55.738150    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:34:57.357636 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:57.357591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:57.358088 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:57.357675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs\") pod \"network-metrics-daemon-7qq7j\" (UID: \"27e17fa7-f7e0-4638-bc66-280734f2d839\") " pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:57.358088 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:57.357775    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:57.358088 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:57.357777    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:57.358088 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:57.357865    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs podName:27e17fa7-f7e0-4638-bc66-280734f2d839 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:13.357833805 +0000 UTC m=+35.919065945 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs") pod "network-metrics-daemon-7qq7j" (UID: "27e17fa7-f7e0-4638-bc66-280734f2d839") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:57.358088 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:57.357881    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret podName:ac1a1089-9a91-4f0c-a285-79e15b4da235 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:13.357874213 +0000 UTC m=+35.919106331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret") pod "global-pull-secret-syncer-dzwbg" (UID: "ac1a1089-9a91-4f0c-a285-79e15b4da235") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:57.458816 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:57.458772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dps2\" (UniqueName: \"kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2\") pod \"network-check-target-5gqw8\" (UID: \"6b68ac73-f537-4ac6-af11-6accaf8431f9\") " pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:57.458998 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:57.458927    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:34:57.458998 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:57.458949    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:34:57.458998 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:57.458963    2559 projected.go:196] Error preparing data for projected volume kube-api-access-4dps2 for pod openshift-network-diagnostics/network-check-target-5gqw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:57.459167 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:57.459026    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2 podName:6b68ac73-f537-4ac6-af11-6accaf8431f9 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:13.45901061 +0000 UTC m=+36.020242751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dps2" (UniqueName: "kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2") pod "network-check-target-5gqw8" (UID: "6b68ac73-f537-4ac6-af11-6accaf8431f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:57.738052 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:57.737964    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:57.738052 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:57.737988    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:57.738271 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:57.738109    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:34:57.738271 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:57.738139    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:57.738378 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:57.738280    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:34:57.738378 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:57.738367    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:34:59.737877 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:59.737824    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:34:59.738270 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:59.737939    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:34:59.738270 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:59.737960    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:34:59.738270 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:34:59.737973    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:34:59.738270 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:59.738084    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:34:59.738270 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:34:59.738107    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:35:00.841810 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.841593    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nrqvf" event={"ID":"51596704-4c57-46c5-a82a-db2ad011ee51","Type":"ContainerStarted","Data":"35aeb8cbb289d9679d11a6fb25642b65ed664b2cbb03f5ea4f9f8a3be660825b"}
Sep 18 15:35:00.843135 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.843113    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn" event={"ID":"3175f4db-dc38-4c58-8def-81d5f23892dc","Type":"ContainerStarted","Data":"bc7d2e6312ff035941b9cc7532c7319682b81314e4adb5f65714d48d3afdc218"}
Sep 18 15:35:00.844591 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.844570    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-94hkp" event={"ID":"e24a42c3-40e3-4986-bd91-2041896deab6","Type":"ContainerStarted","Data":"ca217c4966d9c1b7f0176c966bcd034b0a964d2f9e1a1961c98a2b35d073af1a"}
Sep 18 15:35:00.846043 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.846025    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6bnzx" event={"ID":"60b93ec7-8e32-4d00-a695-ca96c49a0609","Type":"ContainerStarted","Data":"6da53600bad72cc89de0239c96e0e3ffcaebff10083d394f851a3e18a0c88e69"}
Sep 18 15:35:00.847428 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.847411    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fpfzs" event={"ID":"792d0ad5-2007-42a0-bb50-a3b4c0ef1af5","Type":"ContainerStarted","Data":"b0e266f210becdda3b1925f3fb57eaafe43c3f642e5972cbce01ede7f6378257"}
Sep 18 15:35:00.848641 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.848620    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-57xz8" event={"ID":"7e46a8d9-f7b7-469f-99d7-1a13f77741b9","Type":"ContainerStarted","Data":"37373efdd49aad37127a9194d5840f990a4faa41d87f57f208e9a636b190cba5"}
Sep 18 15:35:00.850094 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.850071    2559 generic.go:356] "Generic (PLEG): container finished" podID="9085c12e-916c-423c-a2da-8ba78ec5fc0c" containerID="83fa492d2bc4865b10e0259ec71ca3099ce2adeae66993a30fb341716c7efe61" exitCode=0
Sep 18 15:35:00.850188 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.850159    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnfcs" event={"ID":"9085c12e-916c-423c-a2da-8ba78ec5fc0c","Type":"ContainerDied","Data":"83fa492d2bc4865b10e0259ec71ca3099ce2adeae66993a30fb341716c7efe61"}
Sep 18 15:35:00.855473 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.855451    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct" event={"ID":"4f82eee8-cdc7-4d08-8cee-98827734b5ec","Type":"ContainerStarted","Data":"15f0ee9d93a4107c0a3959abc06bb26590db89c31d7aed7fe56240aa521bdacd"}
Sep 18 15:35:00.855551 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.855536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct" event={"ID":"4f82eee8-cdc7-4d08-8cee-98827734b5ec","Type":"ContainerStarted","Data":"85426bc23cfd6143b2139665a238b58d68b80ce8606716420dffd2ced95ad7a6"}
Sep 18 15:35:00.855586 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.855567    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct" event={"ID":"4f82eee8-cdc7-4d08-8cee-98827734b5ec","Type":"ContainerStarted","Data":"7f4ee343c712085fdc8bd7d82a9c5956612833ce52192c07cf81666e94e9db9e"}
Sep 18 15:35:00.855621 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.855603    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct" event={"ID":"4f82eee8-cdc7-4d08-8cee-98827734b5ec","Type":"ContainerStarted","Data":"5f287c496cf84bd9156971d18774ebd1bec8a6e1ee74ee4f72b8b42921a1e1e5"}
Sep 18 15:35:00.855653 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.855632    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct" event={"ID":"4f82eee8-cdc7-4d08-8cee-98827734b5ec","Type":"ContainerStarted","Data":"0a45d86fcabb751e765db76520488c061b55a2b3511703118021916c230c9736"}
Sep 18 15:35:00.855685 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.855658    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct" event={"ID":"4f82eee8-cdc7-4d08-8cee-98827734b5ec","Type":"ContainerStarted","Data":"8649f20c59396ad8033f4ca0505dc3c513678d041f43e37be60d557aeb9d07f2"}
Sep 18 15:35:00.870326 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.870283    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-223.ec2.internal" podStartSLOduration=19.870269406 podStartE2EDuration="19.870269406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:44.828973573 +0000 UTC m=+7.390205714" watchObservedRunningTime="2026-09-18 15:35:00.870269406 +0000 UTC m=+23.431501557"
Sep 18 15:35:00.870869 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.870827    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nrqvf" podStartSLOduration=4.325608945 podStartE2EDuration="21.870819989s" totalImagesPullingTime="17.545211044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:00.870346356 +0000 UTC m=+23.431578495" watchObservedRunningTime="2026-09-18 15:35:00.870819989 +0000 UTC m=+23.432052129"
Sep 18 15:35:00.954505 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.954449    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-94hkp" podStartSLOduration=4.404091428 podStartE2EDuration="21.954431779s" totalImagesPullingTime="17.550340351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:00.932572182 +0000 UTC m=+23.493804347" watchObservedRunningTime="2026-09-18 15:35:00.954431779 +0000 UTC m=+23.515663919"
Sep 18 15:35:00.954691 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:00.954560    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fpfzs" podStartSLOduration=4.397907296 podStartE2EDuration="21.954551438s" totalImagesPullingTime="17.556644142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:00.954394177 +0000 UTC m=+23.515626342" watchObservedRunningTime="2026-09-18 15:35:00.954551438 +0000 UTC m=+23.515783580"
Sep 18 15:35:01.000070 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.000014    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-57xz8" podStartSLOduration=12.009082532 podStartE2EDuration="21.999997057s" totalImagesPullingTime="9.990914525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:00.975962905 +0000 UTC m=+23.537195055" watchObservedRunningTime="2026-09-18 15:35:00.999997057 +0000 UTC m=+23.561229196"
Sep 18 15:35:01.000491 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.000460    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6bnzx" podStartSLOduration=4.386100448 podStartE2EDuration="22.000449239s" totalImagesPullingTime="17.614348791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:01.000096435 +0000 UTC m=+23.561328600" watchObservedRunningTime="2026-09-18 15:35:01.000449239 +0000 UTC m=+23.561681379"
Sep 18 15:35:01.009918 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.009889    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:01.138288 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.138240    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:01.210037 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.209960    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-57xz8"
Sep 18 15:35:01.210637 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.210618    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-57xz8"
Sep 18 15:35:01.249956 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.249910    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:01.444046 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.443688    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:01.583383 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.583364    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 15:35:01.588494 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.588462    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:01.684237 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.684133    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T15:35:01.58337935Z","UUID":"599d1a78-f4a6-432c-95b9-962692281b47","Handler":null,"Name":"","Endpoint":""}
Sep 18 15:35:01.686000 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.685969    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 15:35:01.686156 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.686024    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 15:35:01.738432 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.738338    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:35:01.738594 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.738338    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:35:01.738594 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:01.738463    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:35:01.738594 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.738338    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:35:01.738594 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:01.738526    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:35:01.738818 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:01.738621    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:35:01.859737 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.859700    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-th4bc" event={"ID":"9026fbbb-e9fa-4be7-8ea9-1fdb1ea8b484","Type":"ContainerStarted","Data":"d95cbdba5b62723706838455028fdf2e9ff555e4ab00b5d6066647def5ed2064"}
Sep 18 15:35:01.861634 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.861608    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn" event={"ID":"3175f4db-dc38-4c58-8def-81d5f23892dc","Type":"ContainerStarted","Data":"3e43cc16345942fe18c029488f03cdaaee5f0a9fa8a0b115404a2b3fb0af7ae6"}
Sep 18 15:35:01.862213 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.862191    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-57xz8"
Sep 18 15:35:01.862314 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.862295    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-57xz8"
Sep 18 15:35:01.903529 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.903471    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-th4bc" podStartSLOduration=5.355331688 podStartE2EDuration="22.903449108s" totalImagesPullingTime="17.54811742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:01.882193916 +0000 UTC m=+24.443426051" watchObservedRunningTime="2026-09-18 15:35:01.903449108 +0000 UTC m=+24.464681248"
Sep 18 15:35:01.979258 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:01.979173    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:02.093446 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:02.093348    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:02.205440 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:02.205322    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:02.869169 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:02.868675    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct" event={"ID":"4f82eee8-cdc7-4d08-8cee-98827734b5ec","Type":"ContainerStarted","Data":"6573717835bfd0fcb286187bf63688340675bbed17bb5058155bbaff15ef9f2f"}
Sep 18 15:35:03.738113 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:03.738077    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:35:03.738307 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:03.738076    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:35:03.738307 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:03.738221    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:35:03.738307 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:03.738258    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:35:03.738307 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:03.738093    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:35:03.738501 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:03.738346    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:35:03.873941 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:03.873904    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn" event={"ID":"3175f4db-dc38-4c58-8def-81d5f23892dc","Type":"ContainerStarted","Data":"d02feacf106b92ae9fbf6cb6469c4b60956ce573cf0f4c16768268bdec262bcd"}
Sep 18 15:35:03.906458 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:03.906385    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nplgn" podStartSLOduration=4.192690236 podStartE2EDuration="23.906361108s" totalImagesPullingTime="19.713670872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:03.904675798 +0000 UTC m=+26.465907938" watchObservedRunningTime="2026-09-18 15:35:03.906361108 +0000 UTC m=+26.467593249"
Sep 18 15:35:05.737847 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:05.737655    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:35:05.738413 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:05.737670    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:35:05.738413 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:05.737945    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:35:05.738413 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:05.737727    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:35:05.738413 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:05.737996    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:35:05.738413 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:05.738137    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:35:05.849900 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:05.849841    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:05.882158 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:05.882125    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct" event={"ID":"4f82eee8-cdc7-4d08-8cee-98827734b5ec","Type":"ContainerStarted","Data":"8f70ae6b4f7a1bde9b781b25caafff6bf42afa1430c4b060350fb5c296fd5d5e"}
Sep 18 15:35:05.882390 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:05.882366    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:35:05.882504 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:05.882435    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:35:05.884164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:05.884137    2559 generic.go:356] "Generic (PLEG): container finished" podID="9085c12e-916c-423c-a2da-8ba78ec5fc0c" containerID="d753caf223d3fd3d1a8c83d8e59f36709ccc18a3c4c61956e587f676dde13e69" exitCode=0
Sep 18 15:35:05.884263 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:05.884178    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnfcs" event={"ID":"9085c12e-916c-423c-a2da-8ba78ec5fc0c","Type":"ContainerDied","Data":"d753caf223d3fd3d1a8c83d8e59f36709ccc18a3c4c61956e587f676dde13e69"}
Sep 18 15:35:05.898088 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:05.898058    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:35:05.925294 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:05.925245    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct" podStartSLOduration=8.371039972 podStartE2EDuration="25.925230923s" totalImagesPullingTime="17.554190951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:05.923567676 +0000 UTC m=+28.484799828" watchObservedRunningTime="2026-09-18 15:35:05.925230923 +0000 UTC m=+28.486463071"
Sep 18 15:35:05.977290 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:05.977251    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:06.886546 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:06.886514    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:35:06.902274 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:06.902244    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:35:07.237330 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:07.237296    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gqw8"]
Sep 18 15:35:07.237477 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:07.237431    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:35:07.237542 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:07.237524    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:35:07.256808 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:07.256778    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dzwbg"]
Sep 18 15:35:07.256969 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:07.256904    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:35:07.257009 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:07.256983    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:35:07.262119 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:07.262093    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7qq7j"]
Sep 18 15:35:07.262233 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:07.262203    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:35:07.262312 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:07.262292    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:35:07.890326 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:07.890287    2559 generic.go:356] "Generic (PLEG): container finished" podID="9085c12e-916c-423c-a2da-8ba78ec5fc0c" containerID="dfc377d09c16b9446c19ddd33b8a38cb1f314dd0cf79dc34de6df442ac2583e6" exitCode=0
Sep 18 15:35:07.890935 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:07.890347    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnfcs" event={"ID":"9085c12e-916c-423c-a2da-8ba78ec5fc0c","Type":"ContainerDied","Data":"dfc377d09c16b9446c19ddd33b8a38cb1f314dd0cf79dc34de6df442ac2583e6"}
Sep 18 15:35:07.973325 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:07.973292    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:08.071939 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:08.071906    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:08.738049 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:08.737849    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:35:08.738226 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:08.737904    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:35:08.738226 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:08.738164    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:35:08.738226 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:08.737913    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:35:08.738226 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:08.738192    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:35:08.738426 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:08.738268    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:35:09.898254 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:09.898218    2559 generic.go:356] "Generic (PLEG): container finished" podID="9085c12e-916c-423c-a2da-8ba78ec5fc0c" containerID="e6b920d8e6ee24688000a91404f5520cb9792038279272b9714bde395a8912e0" exitCode=0
Sep 18 15:35:09.898590 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:09.898287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnfcs" event={"ID":"9085c12e-916c-423c-a2da-8ba78ec5fc0c","Type":"ContainerDied","Data":"e6b920d8e6ee24688000a91404f5520cb9792038279272b9714bde395a8912e0"}
Sep 18 15:35:10.737659 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:10.737621    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:35:10.737659 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:10.737643    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:35:10.737659 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:10.737652    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:35:10.737965 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:10.737767    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:35:10.737965 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:10.737905    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:35:10.738074 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:10.738016    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:35:11.627635 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:11.627596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:11.720437 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:11.720395    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:12.738319 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:12.738288    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:35:12.738732 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:12.738361    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:35:12.738732 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:12.738455    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:35:12.738732 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:12.738460    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzwbg" podUID="ac1a1089-9a91-4f0c-a285-79e15b4da235"
Sep 18 15:35:12.738732 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:12.738566    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qq7j" podUID="27e17fa7-f7e0-4638-bc66-280734f2d839"
Sep 18 15:35:12.738732 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:12.738634    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gqw8" podUID="6b68ac73-f537-4ac6-af11-6accaf8431f9"
Sep 18 15:35:12.895222 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:12.895180    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-223.ec2.internal" event="NodeReady"
Sep 18 15:35:12.895394 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:12.895336    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 15:35:12.968545 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:12.968465    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dw5xs"]
Sep 18 15:35:13.003337 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.003304    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-87jdl"]
Sep 18 15:35:13.003509 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.003474    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dw5xs"
Sep 18 15:35:13.007043 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.007012    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 15:35:13.007186 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.007043    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 15:35:13.007186 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.007052    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 15:35:13.007386 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.007365    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mtwcv\""
Sep 18 15:35:13.025311 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.025280    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dw5xs"]
Sep 18 15:35:13.025461 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.025328    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-87jdl"]
Sep 18 15:35:13.025461 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.025367    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:13.028375 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.028350    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 15:35:13.028509 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.028384    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 15:35:13.028684 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.028672    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9wls\""
Sep 18 15:35:13.078428 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.078392    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f60485ae-76c5-4303-86a2-749c71003198-cert\") pod \"ingress-canary-dw5xs\" (UID: \"f60485ae-76c5-4303-86a2-749c71003198\") " pod="openshift-ingress-canary/ingress-canary-dw5xs"
Sep 18 15:35:13.078622 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.078469    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxt8l\" (UniqueName: \"kubernetes.io/projected/f60485ae-76c5-4303-86a2-749c71003198-kube-api-access-fxt8l\") pod \"ingress-canary-dw5xs\" (UID: \"f60485ae-76c5-4303-86a2-749c71003198\") " pod="openshift-ingress-canary/ingress-canary-dw5xs"
Sep 18 15:35:13.179429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.179392    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st8lq\" (UniqueName: \"kubernetes.io/projected/0e563fb7-63b4-47f8-9c20-5dd15ba96472-kube-api-access-st8lq\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:13.179429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.179450    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxt8l\" (UniqueName: \"kubernetes.io/projected/f60485ae-76c5-4303-86a2-749c71003198-kube-api-access-fxt8l\") pod \"ingress-canary-dw5xs\" (UID: \"f60485ae-76c5-4303-86a2-749c71003198\") " pod="openshift-ingress-canary/ingress-canary-dw5xs"
Sep 18 15:35:13.179707 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.179531    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:13.179707 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.179584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e563fb7-63b4-47f8-9c20-5dd15ba96472-config-volume\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:13.179707 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.179622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e563fb7-63b4-47f8-9c20-5dd15ba96472-tmp-dir\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:13.179707 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.179689    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f60485ae-76c5-4303-86a2-749c71003198-cert\") pod \"ingress-canary-dw5xs\" (UID: \"f60485ae-76c5-4303-86a2-749c71003198\") " pod="openshift-ingress-canary/ingress-canary-dw5xs"
Sep 18 15:35:13.184977 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.184947    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f60485ae-76c5-4303-86a2-749c71003198-cert\") pod \"ingress-canary-dw5xs\" (UID: \"f60485ae-76c5-4303-86a2-749c71003198\") " pod="openshift-ingress-canary/ingress-canary-dw5xs"
Sep 18 15:35:13.199995 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.199965    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxt8l\" (UniqueName: \"kubernetes.io/projected/f60485ae-76c5-4303-86a2-749c71003198-kube-api-access-fxt8l\") pod \"ingress-canary-dw5xs\" (UID: \"f60485ae-76c5-4303-86a2-749c71003198\") " pod="openshift-ingress-canary/ingress-canary-dw5xs"
Sep 18 15:35:13.263397 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.263306    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:13.281083 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.281047    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st8lq\" (UniqueName: \"kubernetes.io/projected/0e563fb7-63b4-47f8-9c20-5dd15ba96472-kube-api-access-st8lq\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:13.281270 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.281104    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:13.281270 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.281131    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e563fb7-63b4-47f8-9c20-5dd15ba96472-config-volume\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:13.281270 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.281174    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e563fb7-63b4-47f8-9c20-5dd15ba96472-tmp-dir\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:13.281270 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:13.281243    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 15:35:13.281419 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:13.281333    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls podName:0e563fb7-63b4-47f8-9c20-5dd15ba96472 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:13.78131026 +0000 UTC m=+36.342542382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls") pod "dns-default-87jdl" (UID: "0e563fb7-63b4-47f8-9c20-5dd15ba96472") : secret "dns-default-metrics-tls" not found
Sep 18 15:35:13.281541 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.281521    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e563fb7-63b4-47f8-9c20-5dd15ba96472-tmp-dir\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:13.281783 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.281752    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e563fb7-63b4-47f8-9c20-5dd15ba96472-config-volume\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:13.297544 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.297508    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st8lq\" (UniqueName: \"kubernetes.io/projected/0e563fb7-63b4-47f8-9c20-5dd15ba96472-kube-api-access-st8lq\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:13.315041 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.315011    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dw5xs"
Sep 18 15:35:13.381772 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.381741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs\") pod \"network-metrics-daemon-7qq7j\" (UID: \"27e17fa7-f7e0-4638-bc66-280734f2d839\") " pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:35:13.381970 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.381784    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:35:13.381970 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:13.381903    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:35:13.381970 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:13.381904    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:35:13.381970 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:13.381969    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret podName:ac1a1089-9a91-4f0c-a285-79e15b4da235 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:45.38194982 +0000 UTC m=+67.943181953 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret") pod "global-pull-secret-syncer-dzwbg" (UID: "ac1a1089-9a91-4f0c-a285-79e15b4da235") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:35:13.382140 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:13.381989    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs podName:27e17fa7-f7e0-4638-bc66-280734f2d839 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:45.381979435 +0000 UTC m=+67.943211556 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs") pod "network-metrics-daemon-7qq7j" (UID: "27e17fa7-f7e0-4638-bc66-280734f2d839") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:35:13.482553 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.482513    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dps2\" (UniqueName: \"kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2\") pod \"network-check-target-5gqw8\" (UID: \"6b68ac73-f537-4ac6-af11-6accaf8431f9\") " pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:35:13.482733 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:13.482694    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:35:13.482733 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:13.482713    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:35:13.482733 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:13.482726    2559 projected.go:196] Error preparing data for projected volume kube-api-access-4dps2 for pod openshift-network-diagnostics/network-check-target-5gqw8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:35:13.482886 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:13.482792    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2 podName:6b68ac73-f537-4ac6-af11-6accaf8431f9 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:45.482771151 +0000 UTC m=+68.044003273 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dps2" (UniqueName: "kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2") pod "network-check-target-5gqw8" (UID: "6b68ac73-f537-4ac6-af11-6accaf8431f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:35:13.501770 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.501733    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:13.785194 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:13.785157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:13.785636 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:13.785295    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 15:35:13.785636 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:13.785354    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls podName:0e563fb7-63b4-47f8-9c20-5dd15ba96472 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:14.785339882 +0000 UTC m=+37.346572000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls") pod "dns-default-87jdl" (UID: "0e563fb7-63b4-47f8-9c20-5dd15ba96472") : secret "dns-default-metrics-tls" not found
Sep 18 15:35:14.376447 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.376252    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dw5xs"]
Sep 18 15:35:14.380304 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:35:14.380273    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60485ae_76c5_4303_86a2_749c71003198.slice/crio-3949482597de3f74f07fbe834cc53dcb9b156d0324beb671bc5cc3851306c344 WatchSource:0}: Error finding container 3949482597de3f74f07fbe834cc53dcb9b156d0324beb671bc5cc3851306c344: Status 404 returned error can't find the container with id 3949482597de3f74f07fbe834cc53dcb9b156d0324beb671bc5cc3851306c344
Sep 18 15:35:14.737743 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.737697    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:35:14.737974 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.737811    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:35:14.737974 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.737887    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:35:14.744531 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.742714    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 15:35:14.744674 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.744583    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 15:35:14.745317 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.744848    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 15:35:14.745317 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.745118    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c572m\""
Sep 18 15:35:14.745317 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.745265    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 15:35:14.746200 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.746179    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nqjbs\""
Sep 18 15:35:14.792785 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.792739    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:14.793278 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:14.792933    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 15:35:14.793278 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:14.793006    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls podName:0e563fb7-63b4-47f8-9c20-5dd15ba96472 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:16.792988887 +0000 UTC m=+39.354221020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls") pod "dns-default-87jdl" (UID: "0e563fb7-63b4-47f8-9c20-5dd15ba96472") : secret "dns-default-metrics-tls" not found
Sep 18 15:35:14.913573 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.913539    2559 generic.go:356] "Generic (PLEG): container finished" podID="9085c12e-916c-423c-a2da-8ba78ec5fc0c" containerID="37804782997fe3e271a1296350998d9664ca3c042e1029a8d5cc77440bcf7e23" exitCode=0
Sep 18 15:35:14.913739 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.913624    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnfcs" event={"ID":"9085c12e-916c-423c-a2da-8ba78ec5fc0c","Type":"ContainerDied","Data":"37804782997fe3e271a1296350998d9664ca3c042e1029a8d5cc77440bcf7e23"}
Sep 18 15:35:14.914865 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:14.914821    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dw5xs" event={"ID":"f60485ae-76c5-4303-86a2-749c71003198","Type":"ContainerStarted","Data":"3949482597de3f74f07fbe834cc53dcb9b156d0324beb671bc5cc3851306c344"}
Sep 18 15:35:15.013647 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:15.013623    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:15.107566 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:15.107532    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:15.577457 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:15.577414    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:15.683596 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:15.683550    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:15.812325 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:15.812289    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:15.917469 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:15.917370    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:15.920029 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:15.920002    2559 generic.go:356] "Generic (PLEG): container finished" podID="9085c12e-916c-423c-a2da-8ba78ec5fc0c" containerID="3a1bbca4ef66a9bd67612c35893570a8535734a3db21df692fa8d33b658df868" exitCode=0
Sep 18 15:35:15.920149 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:15.920045    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnfcs" event={"ID":"9085c12e-916c-423c-a2da-8ba78ec5fc0c","Type":"ContainerDied","Data":"3a1bbca4ef66a9bd67612c35893570a8535734a3db21df692fa8d33b658df868"}
Sep 18 15:35:15.969613 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:15.969580    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4jk9j"]
Sep 18 15:35:15.994395 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:15.994371    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:15.995209 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:15.995187    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4jk9j"]
Sep 18 15:35:16.000351 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.000336    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 15:35:16.000594 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.000581    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 15:35:16.001279 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.001261    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 15:35:16.001337 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.001298    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 15:35:16.001722 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.001704    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c72vd\""
Sep 18 15:35:16.024226 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.024201    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:16.103170 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.103137    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4d75691-4e93-4377-8729-37d2b04c2087-crio-socket\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.103170 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.103172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.103407 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.103276    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zrz\" (UniqueName: \"kubernetes.io/projected/b4d75691-4e93-4377-8729-37d2b04c2087-kube-api-access-c5zrz\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.103407 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.103366    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4d75691-4e93-4377-8729-37d2b04c2087-data-volume\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.103483 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.103412    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.121563 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.121529    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:16.204427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.204334    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.204427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.204420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4d75691-4e93-4377-8729-37d2b04c2087-crio-socket\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.204704 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.204449    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.204704 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.204496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zrz\" (UniqueName: \"kubernetes.io/projected/b4d75691-4e93-4377-8729-37d2b04c2087-kube-api-access-c5zrz\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.204704 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.204562    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4d75691-4e93-4377-8729-37d2b04c2087-data-volume\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.204883 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.204728    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4d75691-4e93-4377-8729-37d2b04c2087-crio-socket\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.204883 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:16.204503    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:16.204883 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:16.204545    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 15:35:16.204883 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:16.204830    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls podName:b4d75691-4e93-4377-8729-37d2b04c2087 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:16.704804024 +0000 UTC m=+39.266036147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4jk9j" (UID: "b4d75691-4e93-4377-8729-37d2b04c2087") : secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:16.205123 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:16.204896    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm podName:b4d75691-4e93-4377-8729-37d2b04c2087 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:16.704877418 +0000 UTC m=+39.266109539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4jk9j" (UID: "b4d75691-4e93-4377-8729-37d2b04c2087") : configmap "kube-rbac-proxy" not found
Sep 18 15:35:16.205123 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.204970    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4d75691-4e93-4377-8729-37d2b04c2087-data-volume\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.227360 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.227330    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zrz\" (UniqueName: \"kubernetes.io/projected/b4d75691-4e93-4377-8729-37d2b04c2087-kube-api-access-c5zrz\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.279401 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.279362    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:16.427530 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.427499    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:16.708023 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.707974    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.708023 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.708033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:16.708288 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:16.708150    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:16.708288 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:16.708175    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 15:35:16.708288 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:16.708235    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls podName:b4d75691-4e93-4377-8729-37d2b04c2087 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:17.708212352 +0000 UTC m=+40.269444484 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4jk9j" (UID: "b4d75691-4e93-4377-8729-37d2b04c2087") : secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:16.708288 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:16.708257    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm podName:b4d75691-4e93-4377-8729-37d2b04c2087 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:17.708246242 +0000 UTC m=+40.269478362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4jk9j" (UID: "b4d75691-4e93-4377-8729-37d2b04c2087") : configmap "kube-rbac-proxy" not found
Sep 18 15:35:16.723557 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.723524    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:16.768360 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.768327    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-4l5h5"]
Sep 18 15:35:16.788300 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.788208    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-4l5h5"]
Sep 18 15:35:16.788452 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.788365    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4l5h5"
Sep 18 15:35:16.791363 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.791329    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4dpq4\""
Sep 18 15:35:16.791509 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.791388    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 15:35:16.792553 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.792531    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 15:35:16.809483 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.809437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:16.809651 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:16.809638    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 15:35:16.809753 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:16.809714    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls podName:0e563fb7-63b4-47f8-9c20-5dd15ba96472 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:20.809696564 +0000 UTC m=+43.370928699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls") pod "dns-default-87jdl" (UID: "0e563fb7-63b4-47f8-9c20-5dd15ba96472") : secret "dns-default-metrics-tls" not found
Sep 18 15:35:16.815564 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.815517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:16.867040 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.867000    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"]
Sep 18 15:35:16.875415 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.875386    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"
Sep 18 15:35:16.878311 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.878284    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-86qzh\""
Sep 18 15:35:16.879117 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.878928    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 15:35:16.879117 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.878936    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 15:35:16.880221 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.880201    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"]
Sep 18 15:35:16.910503 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.910469    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvqq\" (UniqueName: \"kubernetes.io/projected/019d191c-04e0-40df-bfe7-e3c5e53c874d-kube-api-access-rdvqq\") pod \"migrator-76794cbd6c-4l5h5\" (UID: \"019d191c-04e0-40df-bfe7-e3c5e53c874d\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4l5h5"
Sep 18 15:35:16.921700 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.921667    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:16.927138 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.927104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnfcs" event={"ID":"9085c12e-916c-423c-a2da-8ba78ec5fc0c","Type":"ContainerStarted","Data":"30dacad60b6436349c4030ab0914321ec74eaa0a35db7385ac235a30cd93e34b"}
Sep 18 15:35:16.954988 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:16.954922    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lnfcs" podStartSLOduration=9.49031998 podStartE2EDuration="37.954899978s" totalImagesPullingTime="28.464579998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:16.953149119 +0000 UTC m=+39.514381308" watchObservedRunningTime="2026-09-18 15:35:16.954899978 +0000 UTC m=+39.516132136"
Sep 18 15:35:17.011676 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.011638    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-d2g64\" (UID: \"2dcd3e26-1fc0-4356-80a4-fbe8a510822c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"
Sep 18 15:35:17.011845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.011821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvqq\" (UniqueName: \"kubernetes.io/projected/019d191c-04e0-40df-bfe7-e3c5e53c874d-kube-api-access-rdvqq\") pod \"migrator-76794cbd6c-4l5h5\" (UID: \"019d191c-04e0-40df-bfe7-e3c5e53c874d\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4l5h5"
Sep 18 15:35:17.011926 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.011886    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-d2g64\" (UID: \"2dcd3e26-1fc0-4356-80a4-fbe8a510822c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"
Sep 18 15:35:17.022816 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.022784    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvqq\" (UniqueName: \"kubernetes.io/projected/019d191c-04e0-40df-bfe7-e3c5e53c874d-kube-api-access-rdvqq\") pod \"migrator-76794cbd6c-4l5h5\" (UID: \"019d191c-04e0-40df-bfe7-e3c5e53c874d\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4l5h5"
Sep 18 15:35:17.099201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.099112    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4l5h5"
Sep 18 15:35:17.112797 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.112774    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-d2g64\" (UID: \"2dcd3e26-1fc0-4356-80a4-fbe8a510822c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"
Sep 18 15:35:17.112990 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.112967    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-d2g64\" (UID: \"2dcd3e26-1fc0-4356-80a4-fbe8a510822c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"
Sep 18 15:35:17.113122 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:17.113107    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 15:35:17.113190 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:17.113181    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert podName:2dcd3e26-1fc0-4356-80a4-fbe8a510822c nodeName:}" failed. No retries permitted until 2026-09-18 15:35:17.613161312 +0000 UTC m=+40.174393433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-d2g64" (UID: "2dcd3e26-1fc0-4356-80a4-fbe8a510822c") : secret "networking-console-plugin-cert" not found
Sep 18 15:35:17.113743 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.113719    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-d2g64\" (UID: \"2dcd3e26-1fc0-4356-80a4-fbe8a510822c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"
Sep 18 15:35:17.142253 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.142220    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:17.235472 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.235259    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:17.331395 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.331366    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:17.393406 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.393386    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-4l5h5"]
Sep 18 15:35:17.395730 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:35:17.395689    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019d191c_04e0_40df_bfe7_e3c5e53c874d.slice/crio-9a294f95431150942eb7ab5bc245750ea9d356259b5b32a5f1409008c65eeb68 WatchSource:0}: Error finding container 9a294f95431150942eb7ab5bc245750ea9d356259b5b32a5f1409008c65eeb68: Status 404 returned error can't find the container with id 9a294f95431150942eb7ab5bc245750ea9d356259b5b32a5f1409008c65eeb68
Sep 18 15:35:17.616723 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.616640    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-d2g64\" (UID: \"2dcd3e26-1fc0-4356-80a4-fbe8a510822c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"
Sep 18 15:35:17.616910 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:17.616796    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 15:35:17.616910 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:17.616874    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert podName:2dcd3e26-1fc0-4356-80a4-fbe8a510822c nodeName:}" failed. No retries permitted until 2026-09-18 15:35:18.61684586 +0000 UTC m=+41.178077983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-d2g64" (UID: "2dcd3e26-1fc0-4356-80a4-fbe8a510822c") : secret "networking-console-plugin-cert" not found
Sep 18 15:35:17.717949 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.717909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:17.718117 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.718006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:17.718117 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:17.718033    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 15:35:17.718117 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:17.718102    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm podName:b4d75691-4e93-4377-8729-37d2b04c2087 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:19.718086669 +0000 UTC m=+42.279318791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4jk9j" (UID: "b4d75691-4e93-4377-8729-37d2b04c2087") : configmap "kube-rbac-proxy" not found
Sep 18 15:35:17.718117 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:17.718114    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:17.718273 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:17.718154    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls podName:b4d75691-4e93-4377-8729-37d2b04c2087 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:19.718142459 +0000 UTC m=+42.279374577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4jk9j" (UID: "b4d75691-4e93-4377-8729-37d2b04c2087") : secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:17.751636 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.751600    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:17.853563 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.853532    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:17.909634 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.909557    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fpfzs_792d0ad5-2007-42a0-bb50-a3b4c0ef1af5/dns-node-resolver/0.log"
Sep 18 15:35:17.931286 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.931255    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dw5xs" event={"ID":"f60485ae-76c5-4303-86a2-749c71003198","Type":"ContainerStarted","Data":"7dcb21e5eb90042f6ee75e6378638fe3ac444b17841c32a1e7aa09cda80ee65d"}
Sep 18 15:35:17.932427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.932405    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4l5h5" event={"ID":"019d191c-04e0-40df-bfe7-e3c5e53c874d","Type":"ContainerStarted","Data":"9a294f95431150942eb7ab5bc245750ea9d356259b5b32a5f1409008c65eeb68"}
Sep 18 15:35:17.938658 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.938629    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:17.951426 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:17.951382    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dw5xs" podStartSLOduration=3.008916784 podStartE2EDuration="5.951367714s" totalImagesPullingTime="2.94245093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:17.950275187 +0000 UTC m=+40.511507326" watchObservedRunningTime="2026-09-18 15:35:17.951367714 +0000 UTC m=+40.512599853"
Sep 18 15:35:18.224094 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:18.224046    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:18.320448 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:18.320409    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:18.485056 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:18.484947    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:18.625579 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:18.625541    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-d2g64\" (UID: \"2dcd3e26-1fc0-4356-80a4-fbe8a510822c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"
Sep 18 15:35:18.625776 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:18.625713    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 15:35:18.625849 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:18.625807    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert podName:2dcd3e26-1fc0-4356-80a4-fbe8a510822c nodeName:}" failed. No retries permitted until 2026-09-18 15:35:20.625771949 +0000 UTC m=+43.187004068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-d2g64" (UID: "2dcd3e26-1fc0-4356-80a4-fbe8a510822c") : secret "networking-console-plugin-cert" not found
Sep 18 15:35:19.115255 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:19.115228    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nrqvf_51596704-4c57-46c5-a82a-db2ad011ee51/node-ca/0.log"
Sep 18 15:35:19.195586 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:19.195552    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:19.711207 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:19.711170    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dw5xs_f60485ae-76c5-4303-86a2-749c71003198/serve-healthcheck-canary/0.log"
Sep 18 15:35:19.733890 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:19.733850    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:19.734029 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:19.733930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:19.734029 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:19.733994    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:19.734118 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:19.734040    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 15:35:19.734118 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:19.734069    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls podName:b4d75691-4e93-4377-8729-37d2b04c2087 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:23.734052332 +0000 UTC m=+46.295284450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4jk9j" (UID: "b4d75691-4e93-4377-8729-37d2b04c2087") : secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:19.734118 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:19.734088    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm podName:b4d75691-4e93-4377-8729-37d2b04c2087 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:23.734075883 +0000 UTC m=+46.295308000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4jk9j" (UID: "b4d75691-4e93-4377-8729-37d2b04c2087") : configmap "kube-rbac-proxy" not found
Sep 18 15:35:19.942527 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:19.942490    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4l5h5" event={"ID":"019d191c-04e0-40df-bfe7-e3c5e53c874d","Type":"ContainerStarted","Data":"b561c28b23b5e75b3ab48a81952232f8f36c42c867fbfc916d8aa2fe7dc57951"}
Sep 18 15:35:19.942679 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:19.942545    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4l5h5" event={"ID":"019d191c-04e0-40df-bfe7-e3c5e53c874d","Type":"ContainerStarted","Data":"5372b2f8aea99b44edd9ff867b5505b8ca9b3c80a9ff2b609a045a8bdac925e8"}
Sep 18 15:35:19.959481 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:19.959431    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4l5h5" podStartSLOduration=2.4691137530000002 podStartE2EDuration="3.959414657s" totalImagesPullingTime="1.490300904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:19.958850623 +0000 UTC m=+42.520082762" watchObservedRunningTime="2026-09-18 15:35:19.959414657 +0000 UTC m=+42.520646798"
Sep 18 15:35:20.643439 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:20.643394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-d2g64\" (UID: \"2dcd3e26-1fc0-4356-80a4-fbe8a510822c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"
Sep 18 15:35:20.643794 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:20.643522    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 15:35:20.643794 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:20.643580    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert podName:2dcd3e26-1fc0-4356-80a4-fbe8a510822c nodeName:}" failed. No retries permitted until 2026-09-18 15:35:24.643562549 +0000 UTC m=+47.204794670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-d2g64" (UID: "2dcd3e26-1fc0-4356-80a4-fbe8a510822c") : secret "networking-console-plugin-cert" not found
Sep 18 15:35:20.845071 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:20.845033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:20.845242 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:20.845143    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 15:35:20.845242 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:20.845212    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls podName:0e563fb7-63b4-47f8-9c20-5dd15ba96472 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:28.845193989 +0000 UTC m=+51.406426111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls") pod "dns-default-87jdl" (UID: "0e563fb7-63b4-47f8-9c20-5dd15ba96472") : secret "dns-default-metrics-tls" not found
Sep 18 15:35:22.246577 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:22.246541    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:22.465566 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:22.465535    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:22.565620 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:22.565527    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:22.694756 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:22.694727    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:22.794318 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:22.794285    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:22.972067 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:22.972034    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:23.074426 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:23.074397    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:23.768466 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:23.768433    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:23.768928 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:23.768485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:23.768928 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:23.768549    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 15:35:23.768928 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:23.768585    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:23.768928 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:23.768596    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm podName:b4d75691-4e93-4377-8729-37d2b04c2087 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:31.768584079 +0000 UTC m=+54.329816196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4jk9j" (UID: "b4d75691-4e93-4377-8729-37d2b04c2087") : configmap "kube-rbac-proxy" not found
Sep 18 15:35:23.768928 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:23.768701    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls podName:b4d75691-4e93-4377-8729-37d2b04c2087 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:31.768689199 +0000 UTC m=+54.329921317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4jk9j" (UID: "b4d75691-4e93-4377-8729-37d2b04c2087") : secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:23.978426 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:23.978391    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:24.373718 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:24.373681    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:24.564127 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:24.564095    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:24.676109 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:24.676006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-d2g64\" (UID: \"2dcd3e26-1fc0-4356-80a4-fbe8a510822c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"
Sep 18 15:35:24.676258 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:24.676149    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 15:35:24.676258 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:24.676217    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert podName:2dcd3e26-1fc0-4356-80a4-fbe8a510822c nodeName:}" failed. No retries permitted until 2026-09-18 15:35:32.676201905 +0000 UTC m=+55.237434023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-d2g64" (UID: "2dcd3e26-1fc0-4356-80a4-fbe8a510822c") : secret "networking-console-plugin-cert" not found
Sep 18 15:35:24.745811 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:24.745767    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:24.946905 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:24.946844    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:25.954534 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:25.954500    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:26.055160 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:26.055121    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:26.155791 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:26.155760    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:26.340734 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:26.340635    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:26.542875 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:26.542819    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:26.741173 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:26.741134    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:26.801725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:26.801694    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:26.945462 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:26.945427    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:27.941043 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:27.941011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:28.143018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:28.142983    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:28.342294 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:28.342212    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:28.512352 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:28.512320    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:28.572660 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:28.572626    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:28.910472 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:28.910423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:28.913802 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:28.913769    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e563fb7-63b4-47f8-9c20-5dd15ba96472-metrics-tls\") pod \"dns-default-87jdl\" (UID: \"0e563fb7-63b4-47f8-9c20-5dd15ba96472\") " pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:28.935934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:28.935892    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:29.189550 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:29.189521    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:29.204533 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:29.204506    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-87jdl"]
Sep 18 15:35:29.208128 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:35:29.208101    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e563fb7_63b4_47f8_9c20_5dd15ba96472.slice/crio-38bcf185dad44961de076673132b6da526c6e66787290eaec8fc92a439e12a99 WatchSource:0}: Error finding container 38bcf185dad44961de076673132b6da526c6e66787290eaec8fc92a439e12a99: Status 404 returned error can't find the container with id 38bcf185dad44961de076673132b6da526c6e66787290eaec8fc92a439e12a99
Sep 18 15:35:29.333231 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:29.333192    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:29.538878 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:29.538774    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:29.967970 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:29.967931    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:29.968418 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:29.968382    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-87jdl" event={"ID":"0e563fb7-63b4-47f8-9c20-5dd15ba96472","Type":"ContainerStarted","Data":"38bcf185dad44961de076673132b6da526c6e66787290eaec8fc92a439e12a99"}
Sep 18 15:35:30.489933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:30.489892    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:30.591800 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:30.591756    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:31.140747 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:31.140714    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:31.241469 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:31.241388    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:31.344239 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:31.344205    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:31.542228 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:31.542129    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:31.743988 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:31.743953    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:31.830119 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:31.830040    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:31.830119 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:31.830097    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:31.830331 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:31.830192    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:31.830331 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:31.830257    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls podName:b4d75691-4e93-4377-8729-37d2b04c2087 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:47.830241463 +0000 UTC m=+70.391473581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4jk9j" (UID: "b4d75691-4e93-4377-8729-37d2b04c2087") : secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:31.830682 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:31.830655    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4d75691-4e93-4377-8729-37d2b04c2087-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:31.974452 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:31.974412    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-87jdl" event={"ID":"0e563fb7-63b4-47f8-9c20-5dd15ba96472","Type":"ContainerStarted","Data":"4a193a4ea275180513e5ce46466d9180ee0dedaace6620c9db9a855c11ca04c9"}
Sep 18 15:35:31.974602 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:31.974474    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-87jdl" event={"ID":"0e563fb7-63b4-47f8-9c20-5dd15ba96472","Type":"ContainerStarted","Data":"be1577d236be1356816dc323750a2e3e231f21c24aac36ceaeac739cd74d9983"}
Sep 18 15:35:31.974602 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:31.974555    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:31.992187 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:31.992084    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-87jdl" podStartSLOduration=18.209908145 podStartE2EDuration="19.992062499s" totalImagesPullingTime="1.782154354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:31.991837763 +0000 UTC m=+54.553069915" watchObservedRunningTime="2026-09-18 15:35:31.992062499 +0000 UTC m=+54.553294641"
Sep 18 15:35:32.544686 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:32.544648    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:32.738756 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:32.738715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-d2g64\" (UID: \"2dcd3e26-1fc0-4356-80a4-fbe8a510822c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"
Sep 18 15:35:32.741164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:32.741145    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2dcd3e26-1fc0-4356-80a4-fbe8a510822c-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-d2g64\" (UID: \"2dcd3e26-1fc0-4356-80a4-fbe8a510822c\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"
Sep 18 15:35:32.750055 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:32.750025    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:32.787275 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:32.787243    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"
Sep 18 15:35:32.960241 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:32.960205    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:33.079819 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:33.079747    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-d2g64"]
Sep 18 15:35:33.082764 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:35:33.082735    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dcd3e26_1fc0_4356_80a4_fbe8a510822c.slice/crio-70554627bd38caf7d61020f82fb0327b31da58123d2e81b3740808679d42d1b6 WatchSource:0}: Error finding container 70554627bd38caf7d61020f82fb0327b31da58123d2e81b3740808679d42d1b6: Status 404 returned error can't find the container with id 70554627bd38caf7d61020f82fb0327b31da58123d2e81b3740808679d42d1b6
Sep 18 15:35:33.186088 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:33.186034    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:33.288651 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:33.288608    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:33.387875 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:33.387765    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:33.503160 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:33.503124    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:33.678035 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:33.677950    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:33.877889 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:33.877834    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:33.981394 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:33.981356    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64" event={"ID":"2dcd3e26-1fc0-4356-80a4-fbe8a510822c","Type":"ContainerStarted","Data":"70554627bd38caf7d61020f82fb0327b31da58123d2e81b3740808679d42d1b6"}
Sep 18 15:35:33.982567 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:33.982535    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:34.737837 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:34.737794    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:34.853814 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:34.853774    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:34.986201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:34.986169    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64" event={"ID":"2dcd3e26-1fc0-4356-80a4-fbe8a510822c","Type":"ContainerStarted","Data":"2ddbdc3d4fe6a41e49fa8f32225e9e056430c2f6fb01c167e37eaddcae620120"}
Sep 18 15:35:35.014909 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:35.014770    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-d2g64" podStartSLOduration=17.269447322 podStartE2EDuration="19.01474659s" totalImagesPullingTime="1.745299268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:35.013691402 +0000 UTC m=+57.574923566" watchObservedRunningTime="2026-09-18 15:35:35.01474659 +0000 UTC m=+57.575978731"
Sep 18 15:35:35.103525 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:35.103488    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:35.278353 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:35.278270    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:35.381114 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:35.381077    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:35.867032 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:35.866991    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:35.970506 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:35.970467    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:38.911428 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:38.911398    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6qwct"
Sep 18 15:35:40.600844 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.600809    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"]
Sep 18 15:35:40.623589 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.623558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.627491 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.627460    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 15:35:40.627640 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.627618    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 15:35:40.627719 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.627624    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 15:35:40.628820 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.628627    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 15:35:40.629019 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.629001    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 15:35:40.629019 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.629013    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 15:35:40.629205 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.629085    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 15:35:40.634932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.634902    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"]
Sep 18 15:35:40.694025 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.693984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1b00a77f-33af-40fa-af45-1fc6e316a5df-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.694025 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.694026    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b00a77f-33af-40fa-af45-1fc6e316a5df-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.694275 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.694174    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1b00a77f-33af-40fa-af45-1fc6e316a5df-hub\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.694275 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.694216    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1b00a77f-33af-40fa-af45-1fc6e316a5df-ca\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.694275 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.694264    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b00a77f-33af-40fa-af45-1fc6e316a5df-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.694395 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.694305    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jwvm\" (UniqueName: \"kubernetes.io/projected/1b00a77f-33af-40fa-af45-1fc6e316a5df-kube-api-access-6jwvm\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.702283 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.702257    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f987fb898-ndqst"]
Sep 18 15:35:40.702660 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.702633    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:40.714898 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.714875    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.717970 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.717949    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 15:35:40.718075 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.718020    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7tpfr\""
Sep 18 15:35:40.718669 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.718655    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 15:35:40.719303 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.719288    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 15:35:40.726600 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.726582    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 15:35:40.732551 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.732526    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f987fb898-ndqst"]
Sep 18 15:35:40.795634 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795162    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-installation-pull-secrets\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.795634 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795215    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1b00a77f-33af-40fa-af45-1fc6e316a5df-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.795634 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b00a77f-33af-40fa-af45-1fc6e316a5df-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.795634 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795304    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-image-registry-private-configuration\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.795634 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795417    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1b00a77f-33af-40fa-af45-1fc6e316a5df-hub\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.795634 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795461    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1b00a77f-33af-40fa-af45-1fc6e316a5df-ca\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.795634 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795490    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b00a77f-33af-40fa-af45-1fc6e316a5df-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.795634 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795519    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-registry-tls\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.796172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-trusted-ca\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.796172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795693    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-bound-sa-token\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.796172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795727    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jwvm\" (UniqueName: \"kubernetes.io/projected/1b00a77f-33af-40fa-af45-1fc6e316a5df-kube-api-access-6jwvm\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.796172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795776    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-ca-trust-extracted\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.796172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-registry-certificates\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.796172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.795839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-645jl\" (UniqueName: \"kubernetes.io/projected/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-kube-api-access-645jl\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.796172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.796129    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b00a77f-33af-40fa-af45-1fc6e316a5df-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.798200 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.798177    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1b00a77f-33af-40fa-af45-1fc6e316a5df-ca\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.798366 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.798343    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1b00a77f-33af-40fa-af45-1fc6e316a5df-hub\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.798485 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.798468    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b00a77f-33af-40fa-af45-1fc6e316a5df-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.798632 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.798616    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1b00a77f-33af-40fa-af45-1fc6e316a5df-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.810132 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.810105    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jwvm\" (UniqueName: \"kubernetes.io/projected/1b00a77f-33af-40fa-af45-1fc6e316a5df-kube-api-access-6jwvm\") pod \"cluster-proxy-proxy-agent-7f88b46bbb-mxf8p\" (UID: \"1b00a77f-33af-40fa-af45-1fc6e316a5df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.823707 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.823673    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:40.896487 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.896405    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-ca-trust-extracted\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.896487 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.896446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-registry-certificates\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.896487 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.896468    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-645jl\" (UniqueName: \"kubernetes.io/projected/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-kube-api-access-645jl\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.896746 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.896498    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-installation-pull-secrets\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.896746 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.896557    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-image-registry-private-configuration\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.896746 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.896634    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-registry-tls\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.896746 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.896662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-trusted-ca\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.896983 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.896782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-bound-sa-token\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.896983 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.896906    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-ca-trust-extracted\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.897472 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.897405    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-registry-certificates\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.897614 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.897593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-trusted-ca\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.899399 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.899373    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-image-registry-private-configuration\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.899399 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.899391    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-installation-pull-secrets\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.899575 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.899482    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-registry-tls\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.908724 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.908697    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-bound-sa-token\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.909565 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.909539    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-645jl\" (UniqueName: \"kubernetes.io/projected/5f16eb75-a7c7-4904-9b96-a71c1b45dcd7-kube-api-access-645jl\") pod \"image-registry-7f987fb898-ndqst\" (UID: \"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7\") " pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:40.942877 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.942819    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"
Sep 18 15:35:40.992069 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:40.992036    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:41.023188 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:41.023156    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:41.077016 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:41.076982    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:41.241526 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:41.241340    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p"]
Sep 18 15:35:41.244166 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:35:41.244137    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b00a77f_33af_40fa_af45_1fc6e316a5df.slice/crio-bdab8aa9d702aeed6a99e93dd8a71a2a8d39a08af4160335353c011b258b877c WatchSource:0}: Error finding container bdab8aa9d702aeed6a99e93dd8a71a2a8d39a08af4160335353c011b258b877c: Status 404 returned error can't find the container with id bdab8aa9d702aeed6a99e93dd8a71a2a8d39a08af4160335353c011b258b877c
Sep 18 15:35:41.297381 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:41.297314    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:41.319632 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:41.319432    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f987fb898-ndqst"]
Sep 18 15:35:41.321978 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:35:41.321951    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f16eb75_a7c7_4904_9b96_a71c1b45dcd7.slice/crio-4628d26ada5e0d97ca482c6b4c2072d66caae9e39380512ef3bee5b4e335b12b WatchSource:0}: Error finding container 4628d26ada5e0d97ca482c6b4c2072d66caae9e39380512ef3bee5b4e335b12b: Status 404 returned error can't find the container with id 4628d26ada5e0d97ca482c6b4c2072d66caae9e39380512ef3bee5b4e335b12b
Sep 18 15:35:41.395486 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:41.395453    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:41.979669 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:41.979633    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-87jdl"
Sep 18 15:35:42.009083 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:42.009046    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f987fb898-ndqst" event={"ID":"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7","Type":"ContainerStarted","Data":"fa8e8ec4ca7c7ba0cae0ee14aa6f18c12b58ce81b6121cf6b3252c0e270b5cec"}
Sep 18 15:35:42.009265 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:42.009097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f987fb898-ndqst" event={"ID":"5f16eb75-a7c7-4904-9b96-a71c1b45dcd7","Type":"ContainerStarted","Data":"4628d26ada5e0d97ca482c6b4c2072d66caae9e39380512ef3bee5b4e335b12b"}
Sep 18 15:35:42.009265 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:42.009153    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:35:42.010428 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:42.010403    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p" event={"ID":"1b00a77f-33af-40fa-af45-1fc6e316a5df","Type":"ContainerStarted","Data":"bdab8aa9d702aeed6a99e93dd8a71a2a8d39a08af4160335353c011b258b877c"}
Sep 18 15:35:42.030241 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:42.030176    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f987fb898-ndqst" podStartSLOduration=2.030154413 podStartE2EDuration="2.030154413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:42.02915568 +0000 UTC m=+64.590387820" watchObservedRunningTime="2026-09-18 15:35:42.030154413 +0000 UTC m=+64.591386555"
Sep 18 15:35:42.584783 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:42.584732    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:42.684805 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:42.684766    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:42.775667 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:42.775620    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:43.001250 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:43.001188    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:45.023729 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.023696    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p" event={"ID":"1b00a77f-33af-40fa-af45-1fc6e316a5df","Type":"ContainerStarted","Data":"5e3998cdf23a1e83093d675d0f96077ac09df8345c630613a3e45ceffe3b4cde"}
Sep 18 15:35:45.024155 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.023751    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p" event={"ID":"1b00a77f-33af-40fa-af45-1fc6e316a5df","Type":"ContainerStarted","Data":"aef3154adadf223734a206a82c4ab8c3dde3a449c4f282106685d665abb64cc6"}
Sep 18 15:35:45.024155 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.023766    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p" event={"ID":"1b00a77f-33af-40fa-af45-1fc6e316a5df","Type":"ContainerStarted","Data":"620e30f1397568a85846944140fe83368410fdbf7d9199d3263899d8afe2b0cc"}
Sep 18 15:35:45.047037 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.046985    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f88b46bbb-mxf8p" podStartSLOduration=1.446876627 podStartE2EDuration="5.046970909s" totalImagesPullingTime="3.600094282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:45.044687344 +0000 UTC m=+67.605919518" watchObservedRunningTime="2026-09-18 15:35:45.046970909 +0000 UTC m=+67.608203049"
Sep 18 15:35:45.439063 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.439010    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs\") pod \"network-metrics-daemon-7qq7j\" (UID: \"27e17fa7-f7e0-4638-bc66-280734f2d839\") " pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:35:45.439253 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.439080    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:35:45.442092 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.442073    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 15:35:45.442092 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.442084    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 15:35:45.452078 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.452044    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac1a1089-9a91-4f0c-a285-79e15b4da235-original-pull-secret\") pod \"global-pull-secret-syncer-dzwbg\" (UID: \"ac1a1089-9a91-4f0c-a285-79e15b4da235\") " pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:35:45.452202 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.452108    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27e17fa7-f7e0-4638-bc66-280734f2d839-metrics-certs\") pod \"network-metrics-daemon-7qq7j\" (UID: \"27e17fa7-f7e0-4638-bc66-280734f2d839\") " pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:35:45.539628 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.539573    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dps2\" (UniqueName: \"kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2\") pod \"network-check-target-5gqw8\" (UID: \"6b68ac73-f537-4ac6-af11-6accaf8431f9\") " pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:35:45.543018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.542996    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 15:35:45.553691 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.553661    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 15:35:45.563185 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.563156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dps2\" (UniqueName: \"kubernetes.io/projected/6b68ac73-f537-4ac6-af11-6accaf8431f9-kube-api-access-4dps2\") pod \"network-check-target-5gqw8\" (UID: \"6b68ac73-f537-4ac6-af11-6accaf8431f9\") " pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:35:45.652603 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.652573    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nqjbs\""
Sep 18 15:35:45.657463 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.657440    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c572m\""
Sep 18 15:35:45.659493 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.659475    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:35:45.660715 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.660696    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzwbg"
Sep 18 15:35:45.664993 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.664975    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qq7j"
Sep 18 15:35:45.940282 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.940251    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dzwbg"]
Sep 18 15:35:45.942504 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.942481    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gqw8"]
Sep 18 15:35:45.943641 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:35:45.943618    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1a1089_9a91_4f0c_a285_79e15b4da235.slice/crio-0b77c3745c8dd9c63decdc36262e8151d701350e5fd890afb49ff6c8f0cbe224 WatchSource:0}: Error finding container 0b77c3745c8dd9c63decdc36262e8151d701350e5fd890afb49ff6c8f0cbe224: Status 404 returned error can't find the container with id 0b77c3745c8dd9c63decdc36262e8151d701350e5fd890afb49ff6c8f0cbe224
Sep 18 15:35:45.944684 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:35:45.944660    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b68ac73_f537_4ac6_af11_6accaf8431f9.slice/crio-c7fa3bb274b0f7662c01769392ecb2c1507c098d7d1005568bd66ab948ac43ed WatchSource:0}: Error finding container c7fa3bb274b0f7662c01769392ecb2c1507c098d7d1005568bd66ab948ac43ed: Status 404 returned error can't find the container with id c7fa3bb274b0f7662c01769392ecb2c1507c098d7d1005568bd66ab948ac43ed
Sep 18 15:35:45.947382 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:45.947361    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7qq7j"]
Sep 18 15:35:45.950649 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:35:45.950627    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e17fa7_f7e0_4638_bc66_280734f2d839.slice/crio-3075d55d15f77d731469b2f9c0dfaeec08a0d6f4afe82b73fd7b557ae7f5b8ed WatchSource:0}: Error finding container 3075d55d15f77d731469b2f9c0dfaeec08a0d6f4afe82b73fd7b557ae7f5b8ed: Status 404 returned error can't find the container with id 3075d55d15f77d731469b2f9c0dfaeec08a0d6f4afe82b73fd7b557ae7f5b8ed
Sep 18 15:35:46.028355 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:46.028265    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dzwbg" event={"ID":"ac1a1089-9a91-4f0c-a285-79e15b4da235","Type":"ContainerStarted","Data":"0b77c3745c8dd9c63decdc36262e8151d701350e5fd890afb49ff6c8f0cbe224"}
Sep 18 15:35:46.029403 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:46.029380    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qq7j" event={"ID":"27e17fa7-f7e0-4638-bc66-280734f2d839","Type":"ContainerStarted","Data":"3075d55d15f77d731469b2f9c0dfaeec08a0d6f4afe82b73fd7b557ae7f5b8ed"}
Sep 18 15:35:46.030456 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:46.030435    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gqw8" event={"ID":"6b68ac73-f537-4ac6-af11-6accaf8431f9","Type":"ContainerStarted","Data":"c7fa3bb274b0f7662c01769392ecb2c1507c098d7d1005568bd66ab948ac43ed"}
Sep 18 15:35:46.911001 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:46.910964    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:47.011769 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:47.011686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:47.856274 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:47.856220    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:47.859306 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:47.859278    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4d75691-4e93-4377-8729-37d2b04c2087-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4jk9j\" (UID: \"b4d75691-4e93-4377-8729-37d2b04c2087\") " pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:48.040978 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:48.040944    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qq7j" event={"ID":"27e17fa7-f7e0-4638-bc66-280734f2d839","Type":"ContainerStarted","Data":"b3f86f1f37cf01ed1f298b9517f42bfef2ff070d1f81a7ac2f13388be3dfece5"}
Sep 18 15:35:48.041149 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:48.040996    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qq7j" event={"ID":"27e17fa7-f7e0-4638-bc66-280734f2d839","Type":"ContainerStarted","Data":"859502106b20ddcb966b9b203df13579b9e1c67f0b3bdbf60c07308f585d4b3d"}
Sep 18 15:35:48.059575 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:48.059210    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7qq7j" podStartSLOduration=67.958740943 podStartE2EDuration="1m9.059187843s" totalImagesPullingTime="1.1004469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:48.058902297 +0000 UTC m=+70.620134437" watchObservedRunningTime="2026-09-18 15:35:48.059187843 +0000 UTC m=+70.620419984"
Sep 18 15:35:48.123013 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:48.122918    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c72vd\""
Sep 18 15:35:48.130295 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:48.130263    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4jk9j"
Sep 18 15:35:48.162483 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:48.162447    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:48.225990 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:48.225955    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:48.796257 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:48.796217    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:48.891935 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:48.891895    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:48.966506 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:48.966446    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:49.038685 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:49.038648    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:49.345937 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:49.345895    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:49.408804 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:49.408766    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:49.924802 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:49.924752    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:50.024786 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:50.024747    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:50.836185 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:50.835976    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4jk9j"]
Sep 18 15:35:50.838902 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:35:50.838872    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d75691_4e93_4377_8729_37d2b04c2087.slice/crio-abdf03b91d4ce3bf57b1d17e9ecd7fc269acc41e68f978bc080c32e3269de8de WatchSource:0}: Error finding container abdf03b91d4ce3bf57b1d17e9ecd7fc269acc41e68f978bc080c32e3269de8de: Status 404 returned error can't find the container with id abdf03b91d4ce3bf57b1d17e9ecd7fc269acc41e68f978bc080c32e3269de8de
Sep 18 15:35:51.055942 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:51.055902    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gqw8" event={"ID":"6b68ac73-f537-4ac6-af11-6accaf8431f9","Type":"ContainerStarted","Data":"72ca7ec3133901783f2ed5d804539601abd93606205c7f285a6ef9f1bd8b7c56"}
Sep 18 15:35:51.056423 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:51.055992    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:35:51.057465 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:51.057433    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4jk9j" event={"ID":"b4d75691-4e93-4377-8729-37d2b04c2087","Type":"ContainerStarted","Data":"836b619a376492c9569ab099ad6bc836dbb1deddeb175f58aa7a9fbfece9f6aa"}
Sep 18 15:35:51.057584 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:51.057489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4jk9j" event={"ID":"b4d75691-4e93-4377-8729-37d2b04c2087","Type":"ContainerStarted","Data":"abdf03b91d4ce3bf57b1d17e9ecd7fc269acc41e68f978bc080c32e3269de8de"}
Sep 18 15:35:51.058787 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:51.058765    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dzwbg" event={"ID":"ac1a1089-9a91-4f0c-a285-79e15b4da235","Type":"ContainerStarted","Data":"0b2597a61f443dda149afa6c768dcc67f05cbd215a0022533e184c3e7d75d9a3"}
Sep 18 15:35:51.072457 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:51.072406    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5gqw8" podStartSLOduration=66.456006506 podStartE2EDuration="1m11.072391089s" totalImagesPullingTime="4.616384583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:51.071828532 +0000 UTC m=+73.633060685" watchObservedRunningTime="2026-09-18 15:35:51.072391089 +0000 UTC m=+73.633623228"
Sep 18 15:35:51.089049 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:51.088940    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dzwbg" podStartSLOduration=66.461698799 podStartE2EDuration="1m11.088921871s" totalImagesPullingTime="4.627223072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:51.088519303 +0000 UTC m=+73.649751447" watchObservedRunningTime="2026-09-18 15:35:51.088921871 +0000 UTC m=+73.650154010"
Sep 18 15:35:51.500065 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:51.500037    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:51.600586 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:51.600548    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:52.065313 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:52.065254    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4jk9j" event={"ID":"b4d75691-4e93-4377-8729-37d2b04c2087","Type":"ContainerStarted","Data":"d7a7e0fedf42b0fa7dda821c4b159a7830d4d4ca0ceaad2d1133cc5141a0cb33"}
Sep 18 15:35:53.070566 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:53.070478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4jk9j" event={"ID":"b4d75691-4e93-4377-8729-37d2b04c2087","Type":"ContainerStarted","Data":"fff94c0c1c60e3f402af60a9d359f3d4bf67ff2b1d4d376d07546c918cd14f4d"}
Sep 18 15:35:53.097341 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:53.097282    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4jk9j" podStartSLOduration=36.335876382 podStartE2EDuration="38.097266919s" totalImagesPullingTime="1.761390537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:35:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:53.095326526 +0000 UTC m=+75.656558676" watchObservedRunningTime="2026-09-18 15:35:53.097266919 +0000 UTC m=+75.658499073"
Sep 18 15:35:53.559475 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:53.559440    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:53.649651 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:53.649619    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:55.331668 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:55.331630    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:55.499409 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:55.499376    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:55.600542 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:55.600453    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:57.940981 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:57.940947    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:58.060588 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.060555    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:58.211040 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.210951    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:58.316478 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.316441    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:58.357698 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.357662    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9knb9"]
Sep 18 15:35:58.360298 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.360279    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.364320 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.364298    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 15:35:58.364423 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.364305    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 15:35:58.364423 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.364374    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 15:35:58.364423 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.364381    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kpxdd\""
Sep 18 15:35:58.365039 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.365014    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 15:35:58.365144 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.365067    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 15:35:58.366312 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.366293    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 15:35:58.372410 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.372390    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"]
Sep 18 15:35:58.374423 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.374409    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.376951 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.376935    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7hcbh\""
Sep 18 15:35:58.377027 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.376980    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 15:35:58.377551 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.377535    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 15:35:58.378183 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.378169    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 15:35:58.394302 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.394273    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"]
Sep 18 15:35:58.428316 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.428278    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:58.432242 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432223    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab551dde-5363-4914-bb99-c0e65d9ce995-root\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.432324 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432254    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.432324 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432277    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.432427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432347    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-accelerators-collector-config\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.432427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432394    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gks4g\" (UniqueName: \"kubernetes.io/projected/ab551dde-5363-4914-bb99-c0e65d9ce995-kube-api-access-gks4g\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.432539 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432427    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-textfile\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.432539 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432454    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab551dde-5363-4914-bb99-c0e65d9ce995-metrics-client-ca\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.432539 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432491    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab551dde-5363-4914-bb99-c0e65d9ce995-sys\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.432539 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432517    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-wtmp\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.432716 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432573    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.432716 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432608    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-tls\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.432716 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432637    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.432716 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432669    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.432837 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432726    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.432837 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.432750    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7xf\" (UniqueName: \"kubernetes.io/projected/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-api-access-qh7xf\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.496472 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.496384    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:58.533593 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533559    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab551dde-5363-4914-bb99-c0e65d9ce995-metrics-client-ca\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.533593 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab551dde-5363-4914-bb99-c0e65d9ce995-sys\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.533825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533617    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-wtmp\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.533825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533685    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab551dde-5363-4914-bb99-c0e65d9ce995-sys\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.533825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533737    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-wtmp\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.533825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.533825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533768    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-tls\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.533825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533795    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.533825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533818    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.534219 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533837    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.534219 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533887    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7xf\" (UniqueName: \"kubernetes.io/projected/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-api-access-qh7xf\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.534219 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab551dde-5363-4914-bb99-c0e65d9ce995-root\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.534219 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.534219 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.534219 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:58.533963    2559 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 18 15:35:58.534219 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-accelerators-collector-config\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.534219 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.533994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gks4g\" (UniqueName: \"kubernetes.io/projected/ab551dde-5363-4914-bb99-c0e65d9ce995-kube-api-access-gks4g\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.534219 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.534012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-textfile\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.534219 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:35:58.534041    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-state-metrics-tls podName:2bc4f75b-bab6-4f04-9848-f6ff1dd19297 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:59.034018575 +0000 UTC m=+81.595250708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-6wqjh" (UID: "2bc4f75b-bab6-4f04-9848-f6ff1dd19297") : secret "kube-state-metrics-tls" not found
Sep 18 15:35:58.534219 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.534213    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.534735 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.534233    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-textfile\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.534735 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.534330    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab551dde-5363-4914-bb99-c0e65d9ce995-metrics-client-ca\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.534735 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.534550    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab551dde-5363-4914-bb99-c0e65d9ce995-root\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.534842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.534745    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-accelerators-collector-config\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.534973 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.534950    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.535060 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.535022    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.536302 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.536272    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.536396 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.536368    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab551dde-5363-4914-bb99-c0e65d9ce995-node-exporter-tls\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.536732 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.536714    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.546355 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.546332    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7xf\" (UniqueName: \"kubernetes.io/projected/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-api-access-qh7xf\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:58.546462 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.546338    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gks4g\" (UniqueName: \"kubernetes.io/projected/ab551dde-5363-4914-bb99-c0e65d9ce995-kube-api-access-gks4g\") pod \"node-exporter-9knb9\" (UID: \"ab551dde-5363-4914-bb99-c0e65d9ce995\") " pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.618604 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.618562    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:58.669370 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.669331    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9knb9"
Sep 18 15:35:58.679697 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:35:58.679667    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab551dde_5363_4914_bb99_c0e65d9ce995.slice/crio-de0e1431196ea6e2c8d2b50f4fc1577a0cfac9ff894b3351b617649017ed6c59 WatchSource:0}: Error finding container de0e1431196ea6e2c8d2b50f4fc1577a0cfac9ff894b3351b617649017ed6c59: Status 404 returned error can't find the container with id de0e1431196ea6e2c8d2b50f4fc1577a0cfac9ff894b3351b617649017ed6c59
Sep 18 15:35:58.718779 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.718724    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:58.940485 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:58.940452    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:59.038714 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:59.038678    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:59.039203 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:59.039178    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:59.041245 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:59.041220    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc4f75b-bab6-4f04-9848-f6ff1dd19297-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6wqjh\" (UID: \"2bc4f75b-bab6-4f04-9848-f6ff1dd19297\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:59.095040 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:59.095003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9knb9" event={"ID":"ab551dde-5363-4914-bb99-c0e65d9ce995","Type":"ContainerStarted","Data":"de0e1431196ea6e2c8d2b50f4fc1577a0cfac9ff894b3351b617649017ed6c59"}
Sep 18 15:35:59.201107 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:59.200996    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:59.282214 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:59.282172    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"
Sep 18 15:35:59.306625 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:59.306565    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:59.388172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:59.388137    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:35:59.559396 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:59.559334    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-6wqjh"]
Sep 18 15:35:59.562315 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:35:59.562274    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc4f75b_bab6_4f04_9848_f6ff1dd19297.slice/crio-59b06cfb9e8a9f4ad855bf6d041fac094e5a7fdc55cbb47ee7171d0854cb6036 WatchSource:0}: Error finding container 59b06cfb9e8a9f4ad855bf6d041fac094e5a7fdc55cbb47ee7171d0854cb6036: Status 404 returned error can't find the container with id 59b06cfb9e8a9f4ad855bf6d041fac094e5a7fdc55cbb47ee7171d0854cb6036
Sep 18 15:35:59.604520 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:35:59.604472    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:00.100674 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:00.100640    2559 generic.go:356] "Generic (PLEG): container finished" podID="ab551dde-5363-4914-bb99-c0e65d9ce995" containerID="c18e114b177c61af09e019a22db4f810ad679790f43e8a62bdd53918b08203fa" exitCode=0
Sep 18 15:36:00.101139 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:00.100730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9knb9" event={"ID":"ab551dde-5363-4914-bb99-c0e65d9ce995","Type":"ContainerDied","Data":"c18e114b177c61af09e019a22db4f810ad679790f43e8a62bdd53918b08203fa"}
Sep 18 15:36:00.102006 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:00.101986    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh" event={"ID":"2bc4f75b-bab6-4f04-9848-f6ff1dd19297","Type":"ContainerStarted","Data":"59b06cfb9e8a9f4ad855bf6d041fac094e5a7fdc55cbb47ee7171d0854cb6036"}
Sep 18 15:36:01.108562 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:01.108452    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh" event={"ID":"2bc4f75b-bab6-4f04-9848-f6ff1dd19297","Type":"ContainerStarted","Data":"f455d578ba679b2a92032cc8671884e4be6fa2f01f3989ccc55c12326968d675"}
Sep 18 15:36:01.108562 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:01.108531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh" event={"ID":"2bc4f75b-bab6-4f04-9848-f6ff1dd19297","Type":"ContainerStarted","Data":"50686aedce2f7df64ed896533238689fd06f528fa30092c10ccc51c0f9e916ed"}
Sep 18 15:36:01.108562 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:01.108563    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh" event={"ID":"2bc4f75b-bab6-4f04-9848-f6ff1dd19297","Type":"ContainerStarted","Data":"39d62f3b6ffd1ceda8dbc558106b494ce3d1d1a4ebc7d032fa66c706496110cd"}
Sep 18 15:36:01.110576 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:01.110550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9knb9" event={"ID":"ab551dde-5363-4914-bb99-c0e65d9ce995","Type":"ContainerStarted","Data":"9b4568aec891e956d15dbc48f40aa87f2018ae2b828150ffe9188596c05949da"}
Sep 18 15:36:01.110682 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:01.110604    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9knb9" event={"ID":"ab551dde-5363-4914-bb99-c0e65d9ce995","Type":"ContainerStarted","Data":"85d4daea0a60f8ed7635a6ddd1f28c0e8f3f8f734369a89ee5478e14439e4c65"}
Sep 18 15:36:01.130449 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:01.130391    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-6wqjh" podStartSLOduration=1.856019547 podStartE2EDuration="3.130374099s" totalImagesPullingTime="1.274354552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:36:01.128937543 +0000 UTC m=+83.690169683" watchObservedRunningTime="2026-09-18 15:36:01.130374099 +0000 UTC m=+83.691606239"
Sep 18 15:36:01.155210 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:01.155142    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9knb9" podStartSLOduration=1.914902774 podStartE2EDuration="3.155118795s" totalImagesPullingTime="1.177683427s" totalInitContainerRuntime="62.532594ms" isStatefulPod=true podCreationTimestamp="2026-09-18 15:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:36:01.152845602 +0000 UTC m=+83.714077743" watchObservedRunningTime="2026-09-18 15:36:01.155118795 +0000 UTC m=+83.716350936"
Sep 18 15:36:01.315026 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:01.314989    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:01.464364 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:01.464328    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:01.557912 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:01.557878    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:01.909841 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:01.909749    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:03.020142 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:03.020110    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f987fb898-ndqst"
Sep 18 15:36:03.020539 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:03.020176    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:03.141880 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:03.141827    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:03.247445 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:03.247410    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:03.359953 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:03.359835    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:03.456989 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:03.456956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:04.614624 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.614593    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 15:36:04.615647 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.615617    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:04.618422 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.618405    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.621768 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.621746    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 15:36:04.622083 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.622067    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 15:36:04.622409 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.622396    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-98s48\""
Sep 18 15:36:04.622639 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.622620    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 15:36:04.622919 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.622902    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 15:36:04.622982 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.622904    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 15:36:04.623043 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.623026    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 15:36:04.623099 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.623058    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 15:36:04.623158 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.623146    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 15:36:04.623442 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.623418    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 15:36:04.623442 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.623438    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a2ilholjimtpv\""
Sep 18 15:36:04.623608 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.623490    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 15:36:04.623783 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.623767    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 15:36:04.624019 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.623996    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 15:36:04.624115 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.624061    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 15:36:04.625427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.625150    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 15:36:04.628498 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.628481    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 15:36:04.649052 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.649024    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 15:36:04.683188 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683158    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683364 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683196    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683364 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9crg\" (UniqueName: \"kubernetes.io/projected/a32551d5-e289-49b2-a7fd-f584a56ac5b2-kube-api-access-v9crg\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683364 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683245    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683364 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683311    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683486 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683366    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a32551d5-e289-49b2-a7fd-f584a56ac5b2-config-out\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683486 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683392    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683486 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683410    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a32551d5-e289-49b2-a7fd-f584a56ac5b2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683486 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683428    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683486 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-web-config\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683486 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683467    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683659 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683518    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683659 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683556    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683659 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683580    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683659 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683600    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683659 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683615    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683659 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683634    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683672    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-config\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683713    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.683825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.683731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784459 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784416    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784459 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784457    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9crg\" (UniqueName: \"kubernetes.io/projected/a32551d5-e289-49b2-a7fd-f584a56ac5b2-kube-api-access-v9crg\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784686 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784476    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784686 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784686 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a32551d5-e289-49b2-a7fd-f584a56ac5b2-config-out\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784686 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a32551d5-e289-49b2-a7fd-f584a56ac5b2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784727    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-web-config\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784903    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.784932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784934    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.785371 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784959    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.785371 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784981    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.785371 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.785006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-config\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.785371 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.785035    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.785371 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.785063    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.785371 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.785116    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.785653 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.785548    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.785778 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.785753    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.786086 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.786055    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.786963 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.786936    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.787590 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.787568    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.788245 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.788216    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.788356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.784843    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.788356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.788339    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.788600 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.788574    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a32551d5-e289-49b2-a7fd-f584a56ac5b2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.788697 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.788680    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a32551d5-e289-49b2-a7fd-f584a56ac5b2-config-out\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.788789 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.788770    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-web-config\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.789022 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.789000    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.789589 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.789420    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.789727 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.789520    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-config\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.790890 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.790849    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.790989 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.790894    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a32551d5-e289-49b2-a7fd-f584a56ac5b2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.790989 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.790871    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.791185 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.791165    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.791403 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.791387    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a32551d5-e289-49b2-a7fd-f584a56ac5b2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.799001 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.798982    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9crg\" (UniqueName: \"kubernetes.io/projected/a32551d5-e289-49b2-a7fd-f584a56ac5b2-kube-api-access-v9crg\") pod \"prometheus-k8s-0\" (UID: \"a32551d5-e289-49b2-a7fd-f584a56ac5b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:04.928046 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:04.927957    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:05.213332 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:05.213298    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 15:36:05.216132 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:36:05.216101    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32551d5_e289_49b2_a7fd_f584a56ac5b2.slice/crio-3d6516fb3b55545eecd85ebd091bb0ae3fe324df9cb694897c90e4ae5c7a70fa WatchSource:0}: Error finding container 3d6516fb3b55545eecd85ebd091bb0ae3fe324df9cb694897c90e4ae5c7a70fa: Status 404 returned error can't find the container with id 3d6516fb3b55545eecd85ebd091bb0ae3fe324df9cb694897c90e4ae5c7a70fa
Sep 18 15:36:05.816331 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:05.816292    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:05.880644 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:05.880612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:06.132818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:06.132772    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a32551d5-e289-49b2-a7fd-f584a56ac5b2","Type":"ContainerStarted","Data":"3d6516fb3b55545eecd85ebd091bb0ae3fe324df9cb694897c90e4ae5c7a70fa"}
Sep 18 15:36:07.137443 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:07.137409    2559 generic.go:356] "Generic (PLEG): container finished" podID="a32551d5-e289-49b2-a7fd-f584a56ac5b2" containerID="966ffbe33036a8afe60b253e72801a9dbedfe962d73fc0ad7a57885652345834" exitCode=0
Sep 18 15:36:07.137876 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:07.137499    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a32551d5-e289-49b2-a7fd-f584a56ac5b2","Type":"ContainerDied","Data":"966ffbe33036a8afe60b253e72801a9dbedfe962d73fc0ad7a57885652345834"}
Sep 18 15:36:11.155902 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:11.155848    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a32551d5-e289-49b2-a7fd-f584a56ac5b2","Type":"ContainerStarted","Data":"f549f5c5bc22a05f71ab7746f1347bc8f0c9207b98cbff777c7aeed9e99a3884"}
Sep 18 15:36:11.156344 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:11.155953    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a32551d5-e289-49b2-a7fd-f584a56ac5b2","Type":"ContainerStarted","Data":"4fb95b51c761424d4d39bd3a0d12adf020de2444c99e7d312119610ef8d50be2"}
Sep 18 15:36:11.212667 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:11.212628    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:11.296690 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:11.296656    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:13.166600 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:13.166563    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a32551d5-e289-49b2-a7fd-f584a56ac5b2","Type":"ContainerStarted","Data":"1debea2dd0885c3e0004bfa6858a7f1b5860856b511c02f9ead7981b10182f98"}
Sep 18 15:36:13.167077 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:13.166629    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a32551d5-e289-49b2-a7fd-f584a56ac5b2","Type":"ContainerStarted","Data":"3ce680fe7f751917a005531bbcec3bb89e62820fb0be8deeaf2a5626b172d4e7"}
Sep 18 15:36:13.167077 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:13.166657    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a32551d5-e289-49b2-a7fd-f584a56ac5b2","Type":"ContainerStarted","Data":"3fe2a9e0a3ecd15a55225de60af19f0f37f6a0d9de3160c549e66af052a62f2d"}
Sep 18 15:36:13.167077 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:13.166680    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a32551d5-e289-49b2-a7fd-f584a56ac5b2","Type":"ContainerStarted","Data":"92d944eb790c80e6437d2381e4993dcb49e726e3ab5ce6cd84475e5d54e0e4a4"}
Sep 18 15:36:13.206473 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:13.206411    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.368863908 podStartE2EDuration="9.206390152s" totalImagesPullingTime="5.819198403s" totalInitContainerRuntime="18.327841ms" isStatefulPod=false podCreationTimestamp="2026-09-18 15:36:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:36:13.204246617 +0000 UTC m=+95.765478854" watchObservedRunningTime="2026-09-18 15:36:13.206390152 +0000 UTC m=+95.767622311"
Sep 18 15:36:14.125633 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:14.125578    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:14.226022 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:14.225982    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:14.326812 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:14.326774    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:14.928589 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:14.928550    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:36:20.815121 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:20.815079    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:20.887987 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:20.887947    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:22.067961 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:22.067934    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5gqw8"
Sep 18 15:36:23.028442 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:23.028401    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:23.089178 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:23.089138    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:30.139889 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:30.139834    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:36:30.209894 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:36:30.209840    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:04.928263 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:04.928227    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:37:04.949180 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:04.949138    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:37:05.370513 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:05.370485    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 15:37:17.622609 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:17.622567    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:18.053952 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:18.053894    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:25.468391 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:25.468297    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:25.533419 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:25.533381    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:32.896928 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:32.896879    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:32.967705 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:32.967663    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:33.617463 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:33.617425    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:33.676292 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:33.676258    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:34.405378 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:34.405336    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:34.506647 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:34.506612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:44.505781 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:44.505748    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:44.606905 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:44.606874    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:44.659753 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:44.659721    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:44.739490 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:44.739449    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:44.842262 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:44.842176    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:50.798972 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:50.798926    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:50.898237 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:50.898198    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:50.956873 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:50.956816    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:56.825867 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:56.825817    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:37:56.883792 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:37:56.883757    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:39:11.746187 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:39:11.746086    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:39:39.637503 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:39:39.637478    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 15:39:52.693636 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:39:52.693594    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:39:54.326767 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:39:54.326714    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:14.784193 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:14.784151    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:15.630963 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:15.630927    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:18.423193 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:18.423155    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:18.524511 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:18.524481    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:20.301557 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:20.301524    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:20.448400 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:20.448367    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:20.589050 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:20.588968    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:20.695980 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:20.695950    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:22.920244 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:22.920168    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:23.119108 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:23.119073    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:25.191960 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:25.191931    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:25.259019 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:25.258988    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:30.500212 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:30.500171    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:31.036078 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:31.036021    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:31.126876 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:31.126817    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:32.938432 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:32.938385    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:33.039502 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:33.039467    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:37.660822 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:37.660777    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:37.722715 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:37.722677    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:40.272496 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:40.272452    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:45.496795 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:45.496757    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:45.690139 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:45.690095    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:46.245706 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:46.245667    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:46.346875 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:46.346821    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:56.303872 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:56.303821    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:56.386016 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:56.385987    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:56.488617 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:56.488584    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:56.589903 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:56.589811    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:57.062285 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.062252    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:57.291961 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.291922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:57.391277 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.391178    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:57.444478 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.444448    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"]
Sep 18 15:40:57.447708 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.447691    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.451099 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.451080    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 15:40:57.451460 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.451443    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 15:40:57.451550 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.451536    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 15:40:57.451702 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.451677    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cqmrf\""
Sep 18 15:40:57.451909 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.451897    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 15:40:57.466512 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.466486    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"]
Sep 18 15:40:57.466880 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.466839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e64e9a8-1885-4a01-a361-241cbe4b96d2-tmp\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.467005 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.466890    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3e64e9a8-1885-4a01-a361-241cbe4b96d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.467005 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.466957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e64e9a8-1885-4a01-a361-241cbe4b96d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.467005 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.466999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dshsw\" (UniqueName: \"kubernetes.io/projected/3e64e9a8-1885-4a01-a361-241cbe4b96d2-kube-api-access-dshsw\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.467138 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.467041    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e64e9a8-1885-4a01-a361-241cbe4b96d2-manifests\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.567373 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.567343    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dshsw\" (UniqueName: \"kubernetes.io/projected/3e64e9a8-1885-4a01-a361-241cbe4b96d2-kube-api-access-dshsw\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.567516 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.567379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e64e9a8-1885-4a01-a361-241cbe4b96d2-manifests\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.567516 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.567426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e64e9a8-1885-4a01-a361-241cbe4b96d2-tmp\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.567516 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.567444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3e64e9a8-1885-4a01-a361-241cbe4b96d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.567516 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.567462    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e64e9a8-1885-4a01-a361-241cbe4b96d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.567799 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.567780    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e64e9a8-1885-4a01-a361-241cbe4b96d2-manifests\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.569815 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.569787    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e64e9a8-1885-4a01-a361-241cbe4b96d2-tmp\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.569966 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.569932    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e64e9a8-1885-4a01-a361-241cbe4b96d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.570046 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.570030    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3e64e9a8-1885-4a01-a361-241cbe4b96d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.577333 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.577312    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dshsw\" (UniqueName: \"kubernetes.io/projected/3e64e9a8-1885-4a01-a361-241cbe4b96d2-kube-api-access-dshsw\") pod \"opendatahub-operator-controller-manager-54b8c8d665-pm8h9\" (UID: \"3e64e9a8-1885-4a01-a361-241cbe4b96d2\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.588149 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.588124    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:57.687759 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.687735    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:57.757960 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.757937    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:40:57.800106 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.800068    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:57.888163 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.888133    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:57.990263 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:57.990144    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:58.040615 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:58.038378    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"]
Sep 18 15:40:58.042250 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:40:58.042217    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e64e9a8_1885_4a01_a361_241cbe4b96d2.slice/crio-fea70c218fe4f039a93c5663b8967c708ee71b8d0653b1a2c000ab67e0e859e1 WatchSource:0}: Error finding container fea70c218fe4f039a93c5663b8967c708ee71b8d0653b1a2c000ab67e0e859e1: Status 404 returned error can't find the container with id fea70c218fe4f039a93c5663b8967c708ee71b8d0653b1a2c000ab67e0e859e1
Sep 18 15:40:58.044528 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:58.044512    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 15:40:58.089498 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:58.089475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:58.190505 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:58.190462    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:58.194524 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:58.194500    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9" event={"ID":"3e64e9a8-1885-4a01-a361-241cbe4b96d2","Type":"ContainerStarted","Data":"fea70c218fe4f039a93c5663b8967c708ee71b8d0653b1a2c000ab67e0e859e1"}
Sep 18 15:40:58.291100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:58.291037    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:40:58.400984 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:40:58.400954    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:01.209249 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:01.209215    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9" event={"ID":"3e64e9a8-1885-4a01-a361-241cbe4b96d2","Type":"ContainerStarted","Data":"10e6bcdc01929426691b6f6161bdf0acd4febf1ba8e550f54cfc48c53f47210b"}
Sep 18 15:41:02.214217 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:02.214185    2559 generic.go:356] "Generic (PLEG): container finished" podID="3e64e9a8-1885-4a01-a361-241cbe4b96d2" containerID="10e6bcdc01929426691b6f6161bdf0acd4febf1ba8e550f54cfc48c53f47210b" exitCode=0
Sep 18 15:41:02.214573 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:02.214268    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9" event={"ID":"3e64e9a8-1885-4a01-a361-241cbe4b96d2","Type":"ContainerDied","Data":"10e6bcdc01929426691b6f6161bdf0acd4febf1ba8e550f54cfc48c53f47210b"}
Sep 18 15:41:02.549137 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:02.549089    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:02.654769 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:02.654738    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:03.219913 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:03.219884    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9" event={"ID":"3e64e9a8-1885-4a01-a361-241cbe4b96d2","Type":"ContainerStarted","Data":"038b033b069d3977e0003c85aa4394148b41cdcde20551b333cb0b1e9726f0a1"}
Sep 18 15:41:03.220296 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:03.219982    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:41:03.266096 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:03.266044    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9" podStartSLOduration=1.930824203 podStartE2EDuration="6.26602771s" totalImagesPullingTime="3.32606503s" totalInitContainerRuntime="1.009138477s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:40:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:41:03.263659562 +0000 UTC m=+385.824891715" watchObservedRunningTime="2026-09-18 15:41:03.26602771 +0000 UTC m=+385.827259830"
Sep 18 15:41:09.065652 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:09.065616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:09.151295 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:09.151264    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:14.225727 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:14.225700    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-pm8h9"
Sep 18 15:41:17.146171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:17.146135    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:17.364758 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:17.364724    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:17.664114 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:17.664072    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:17.841242 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:17.841194    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:19.368176 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:19.368138    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:19.453065 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:19.453030    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:20.657697 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:20.657636    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:24.434060 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:24.434021    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:24.532610 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:24.532569    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:25.128478 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:25.128445    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:25.229685 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:25.229640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:25.328407 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:25.328366    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:28.229678 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:28.229636    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:28.329991 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:28.329956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:28.433241 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:28.433207    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:28.532580 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:28.532489    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:28.843839 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:28.843747    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:29.031871 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:29.031818    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:36.181042 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.181002    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"]
Sep 18 15:41:36.185579 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.185557    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.189267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.189237    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 15:41:36.190339 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.190319    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 15:41:36.190451 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.190319    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 15:41:36.190451 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.190399    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 15:41:36.190608 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.190594    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 15:41:36.191713 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.191697    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jspcm\""
Sep 18 15:41:36.199470 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.199439    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"]
Sep 18 15:41:36.269747 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.269705    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/21c9275d-e252-4185-91db-aceedf69e3a0-manager-config\") pod \"lws-controller-manager-c59846bb8-qwrhp\" (UID: \"21c9275d-e252-4185-91db-aceedf69e3a0\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.269747 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.269750    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21c9275d-e252-4185-91db-aceedf69e3a0-metrics-cert\") pod \"lws-controller-manager-c59846bb8-qwrhp\" (UID: \"21c9275d-e252-4185-91db-aceedf69e3a0\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.270029 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.269774    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21c9275d-e252-4185-91db-aceedf69e3a0-cert\") pod \"lws-controller-manager-c59846bb8-qwrhp\" (UID: \"21c9275d-e252-4185-91db-aceedf69e3a0\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.270029 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.269839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fnzg\" (UniqueName: \"kubernetes.io/projected/21c9275d-e252-4185-91db-aceedf69e3a0-kube-api-access-6fnzg\") pod \"lws-controller-manager-c59846bb8-qwrhp\" (UID: \"21c9275d-e252-4185-91db-aceedf69e3a0\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.371324 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.371284    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/21c9275d-e252-4185-91db-aceedf69e3a0-manager-config\") pod \"lws-controller-manager-c59846bb8-qwrhp\" (UID: \"21c9275d-e252-4185-91db-aceedf69e3a0\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.371324 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.371326    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21c9275d-e252-4185-91db-aceedf69e3a0-metrics-cert\") pod \"lws-controller-manager-c59846bb8-qwrhp\" (UID: \"21c9275d-e252-4185-91db-aceedf69e3a0\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.371580 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.371460    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21c9275d-e252-4185-91db-aceedf69e3a0-cert\") pod \"lws-controller-manager-c59846bb8-qwrhp\" (UID: \"21c9275d-e252-4185-91db-aceedf69e3a0\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.371580 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.371524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fnzg\" (UniqueName: \"kubernetes.io/projected/21c9275d-e252-4185-91db-aceedf69e3a0-kube-api-access-6fnzg\") pod \"lws-controller-manager-c59846bb8-qwrhp\" (UID: \"21c9275d-e252-4185-91db-aceedf69e3a0\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.372006 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.371972    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/21c9275d-e252-4185-91db-aceedf69e3a0-manager-config\") pod \"lws-controller-manager-c59846bb8-qwrhp\" (UID: \"21c9275d-e252-4185-91db-aceedf69e3a0\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.373987 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.373960    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21c9275d-e252-4185-91db-aceedf69e3a0-metrics-cert\") pod \"lws-controller-manager-c59846bb8-qwrhp\" (UID: \"21c9275d-e252-4185-91db-aceedf69e3a0\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.373987 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.373968    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21c9275d-e252-4185-91db-aceedf69e3a0-cert\") pod \"lws-controller-manager-c59846bb8-qwrhp\" (UID: \"21c9275d-e252-4185-91db-aceedf69e3a0\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.381018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.380986    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fnzg\" (UniqueName: \"kubernetes.io/projected/21c9275d-e252-4185-91db-aceedf69e3a0-kube-api-access-6fnzg\") pod \"lws-controller-manager-c59846bb8-qwrhp\" (UID: \"21c9275d-e252-4185-91db-aceedf69e3a0\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.495187 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.495094    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:36.770104 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:36.769982    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"]
Sep 18 15:41:36.773442 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:41:36.773412    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c9275d_e252_4185_91db_aceedf69e3a0.slice/crio-51c2153ec6215a89b432d4209cc73ea15945e2e291264e9bfa2195ea6f12e8c6 WatchSource:0}: Error finding container 51c2153ec6215a89b432d4209cc73ea15945e2e291264e9bfa2195ea6f12e8c6: Status 404 returned error can't find the container with id 51c2153ec6215a89b432d4209cc73ea15945e2e291264e9bfa2195ea6f12e8c6
Sep 18 15:41:37.346987 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:37.346948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp" event={"ID":"21c9275d-e252-4185-91db-aceedf69e3a0","Type":"ContainerStarted","Data":"51c2153ec6215a89b432d4209cc73ea15945e2e291264e9bfa2195ea6f12e8c6"}
Sep 18 15:41:38.167939 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.167900    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:38.286368 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.286334    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"]
Sep 18 15:41:38.289626 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.289602    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"
Sep 18 15:41:38.292558 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.292533    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 18 15:41:38.292692 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.292641    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-44tqm\""
Sep 18 15:41:38.312106 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.312071    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"]
Sep 18 15:41:38.313934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.313900    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"]
Sep 18 15:41:38.317512 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.317491    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"
Sep 18 15:41:38.320467 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.320446    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-h5rtk\""
Sep 18 15:41:38.320936 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.320919    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 18 15:41:38.335087 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.335055    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"]
Sep 18 15:41:38.343486 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.343456    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:38.389397 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.389348    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j66l\" (UniqueName: \"kubernetes.io/projected/92275e0d-82a3-472b-88ca-d630ba4f36da-kube-api-access-8j66l\") pod \"llmisvc-controller-manager-777c6bd7d5-hbfxb\" (UID: \"92275e0d-82a3-472b-88ca-d630ba4f36da\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"
Sep 18 15:41:38.389920 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.389410    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrzc9\" (UniqueName: \"kubernetes.io/projected/0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302-kube-api-access-lrzc9\") pod \"kserve-controller-manager-d6f4b65b8-n5q6z\" (UID: \"0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"
Sep 18 15:41:38.389920 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.389459    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302-cert\") pod \"kserve-controller-manager-d6f4b65b8-n5q6z\" (UID: \"0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"
Sep 18 15:41:38.389920 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.389489    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92275e0d-82a3-472b-88ca-d630ba4f36da-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hbfxb\" (UID: \"92275e0d-82a3-472b-88ca-d630ba4f36da\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"
Sep 18 15:41:38.490761 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.490723    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrzc9\" (UniqueName: \"kubernetes.io/projected/0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302-kube-api-access-lrzc9\") pod \"kserve-controller-manager-d6f4b65b8-n5q6z\" (UID: \"0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"
Sep 18 15:41:38.490967 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.490776    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302-cert\") pod \"kserve-controller-manager-d6f4b65b8-n5q6z\" (UID: \"0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"
Sep 18 15:41:38.490967 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.490808    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92275e0d-82a3-472b-88ca-d630ba4f36da-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hbfxb\" (UID: \"92275e0d-82a3-472b-88ca-d630ba4f36da\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"
Sep 18 15:41:38.490967 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.490868    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j66l\" (UniqueName: \"kubernetes.io/projected/92275e0d-82a3-472b-88ca-d630ba4f36da-kube-api-access-8j66l\") pod \"llmisvc-controller-manager-777c6bd7d5-hbfxb\" (UID: \"92275e0d-82a3-472b-88ca-d630ba4f36da\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"
Sep 18 15:41:38.490967 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:41:38.490944    2559 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 18 15:41:38.491143 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:41:38.490978    2559 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 18 15:41:38.491143 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:41:38.491017    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302-cert podName:0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302 nodeName:}" failed. No retries permitted until 2026-09-18 15:41:38.990997589 +0000 UTC m=+421.552229721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302-cert") pod "kserve-controller-manager-d6f4b65b8-n5q6z" (UID: "0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302") : secret "kserve-webhook-server-cert" not found
Sep 18 15:41:38.491143 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:41:38.491036    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92275e0d-82a3-472b-88ca-d630ba4f36da-cert podName:92275e0d-82a3-472b-88ca-d630ba4f36da nodeName:}" failed. No retries permitted until 2026-09-18 15:41:38.991027308 +0000 UTC m=+421.552259431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/92275e0d-82a3-472b-88ca-d630ba4f36da-cert") pod "llmisvc-controller-manager-777c6bd7d5-hbfxb" (UID: "92275e0d-82a3-472b-88ca-d630ba4f36da") : secret "llmisvc-webhook-server-cert" not found
Sep 18 15:41:38.506386 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.506355    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrzc9\" (UniqueName: \"kubernetes.io/projected/0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302-kube-api-access-lrzc9\") pod \"kserve-controller-manager-d6f4b65b8-n5q6z\" (UID: \"0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"
Sep 18 15:41:38.507971 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.507935    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j66l\" (UniqueName: \"kubernetes.io/projected/92275e0d-82a3-472b-88ca-d630ba4f36da-kube-api-access-8j66l\") pod \"llmisvc-controller-manager-777c6bd7d5-hbfxb\" (UID: \"92275e0d-82a3-472b-88ca-d630ba4f36da\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"
Sep 18 15:41:38.691412 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.691356    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:38.846332 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.846220    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:38.995516 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.995473    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302-cert\") pod \"kserve-controller-manager-d6f4b65b8-n5q6z\" (UID: \"0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"
Sep 18 15:41:38.995698 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.995536    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92275e0d-82a3-472b-88ca-d630ba4f36da-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hbfxb\" (UID: \"92275e0d-82a3-472b-88ca-d630ba4f36da\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"
Sep 18 15:41:38.998213 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.998182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302-cert\") pod \"kserve-controller-manager-d6f4b65b8-n5q6z\" (UID: \"0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"
Sep 18 15:41:38.998343 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:38.998182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92275e0d-82a3-472b-88ca-d630ba4f36da-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hbfxb\" (UID: \"92275e0d-82a3-472b-88ca-d630ba4f36da\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"
Sep 18 15:41:39.202038 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:39.201991    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"
Sep 18 15:41:39.228628 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:39.228589    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"
Sep 18 15:41:39.567576 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:39.567536    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:39.627117 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:39.627092    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"]
Sep 18 15:41:39.628806 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:39.628781    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"]
Sep 18 15:41:39.629076 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:41:39.629054    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod92275e0d_82a3_472b_88ca_d630ba4f36da.slice/crio-cbc5f0f02ac5ded3854b3ba0741def626c6e9e0dc3aac1b3029ab214d66bff0c WatchSource:0}: Error finding container cbc5f0f02ac5ded3854b3ba0741def626c6e9e0dc3aac1b3029ab214d66bff0c: Status 404 returned error can't find the container with id cbc5f0f02ac5ded3854b3ba0741def626c6e9e0dc3aac1b3029ab214d66bff0c
Sep 18 15:41:39.630324 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:41:39.630292    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fdb8fa7_270e_45a4_8ee5_1a0a4b6c2302.slice/crio-3ed17c9a768c868fcc884dc683015435ab39116f75ca161c66902c5e4406e7a9 WatchSource:0}: Error finding container 3ed17c9a768c868fcc884dc683015435ab39116f75ca161c66902c5e4406e7a9: Status 404 returned error can't find the container with id 3ed17c9a768c868fcc884dc683015435ab39116f75ca161c66902c5e4406e7a9
Sep 18 15:41:39.670075 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:39.670022    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:41:40.365183 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:40.365140    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z" event={"ID":"0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302","Type":"ContainerStarted","Data":"3ed17c9a768c868fcc884dc683015435ab39116f75ca161c66902c5e4406e7a9"}
Sep 18 15:41:40.367199 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:40.367164    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp" event={"ID":"21c9275d-e252-4185-91db-aceedf69e3a0","Type":"ContainerStarted","Data":"a66cb70169b2a16181c1f49e8235c9b9edd8cbcd12f462a54e9d3ff7e28377d6"}
Sep 18 15:41:40.367961 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:40.367937    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:41:40.370071 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:40.370044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb" event={"ID":"92275e0d-82a3-472b-88ca-d630ba4f36da","Type":"ContainerStarted","Data":"cbc5f0f02ac5ded3854b3ba0741def626c6e9e0dc3aac1b3029ab214d66bff0c"}
Sep 18 15:41:40.389570 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:40.389503    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp" podStartSLOduration=1.754282567 podStartE2EDuration="4.389479643s" totalImagesPullingTime="2.635197076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:41:40.386377081 +0000 UTC m=+422.947609221" watchObservedRunningTime="2026-09-18 15:41:40.389479643 +0000 UTC m=+422.950711802"
Sep 18 15:41:43.386243 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:43.386139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z" event={"ID":"0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302","Type":"ContainerStarted","Data":"00cf2958060a318f10a72b082d0adf0d4d52dc73a489432a4088f5d981179779"}
Sep 18 15:41:43.386243 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:43.386213    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"
Sep 18 15:41:43.388157 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:43.388125    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb" event={"ID":"92275e0d-82a3-472b-88ca-d630ba4f36da","Type":"ContainerStarted","Data":"afb6b998782581c6525d2a7ae494df4cc932eb6f25859ea20e725fe8fa726fa4"}
Sep 18 15:41:43.388347 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:43.388330    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"
Sep 18 15:41:43.413837 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:43.413783    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z" podStartSLOduration=1.96828008 podStartE2EDuration="5.413766446s" totalImagesPullingTime="3.445486366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:41:43.41122645 +0000 UTC m=+425.972458589" watchObservedRunningTime="2026-09-18 15:41:43.413766446 +0000 UTC m=+425.974998586"
Sep 18 15:41:43.444186 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:43.444127    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb" podStartSLOduration=1.997753465 podStartE2EDuration="5.444109957s" totalImagesPullingTime="3.446356492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:41:43.443811512 +0000 UTC m=+426.005043654" watchObservedRunningTime="2026-09-18 15:41:43.444109957 +0000 UTC m=+426.005342097"
Sep 18 15:41:52.382039 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:41:52.381951    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-qwrhp"
Sep 18 15:42:14.394430 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:14.394395    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hbfxb"
Sep 18 15:42:14.397138 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:14.397115    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5q6z"
Sep 18 15:42:14.547339 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:14.547295    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:29.143293 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:29.143254    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:29.236376 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:29.236343    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:38.281058 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:38.281013    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:42.437999 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:42.437966    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:44.820379 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:44.820344    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:45.211600 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:45.211556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:46.965294 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:46.965260    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zqqmm"]
Sep 18 15:42:46.967464 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:46.967448    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-zqqmm"
Sep 18 15:42:46.970275 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:46.970250    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 15:42:46.971325 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:46.971301    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-m7j2x\""
Sep 18 15:42:46.971433 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:46.971306    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 15:42:46.983850 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:46.983828    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zqqmm"]
Sep 18 15:42:47.082305 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:47.082269    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sncr2\" (UniqueName: \"kubernetes.io/projected/2f1e38d2-8012-49ce-bdc4-de7d55d11239-kube-api-access-sncr2\") pod \"authorino-operator-76b478bd78-zqqmm\" (UID: \"2f1e38d2-8012-49ce-bdc4-de7d55d11239\") " pod="kuadrant-system/authorino-operator-76b478bd78-zqqmm"
Sep 18 15:42:47.182703 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:47.182668    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sncr2\" (UniqueName: \"kubernetes.io/projected/2f1e38d2-8012-49ce-bdc4-de7d55d11239-kube-api-access-sncr2\") pod \"authorino-operator-76b478bd78-zqqmm\" (UID: \"2f1e38d2-8012-49ce-bdc4-de7d55d11239\") " pod="kuadrant-system/authorino-operator-76b478bd78-zqqmm"
Sep 18 15:42:47.203807 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:47.203773    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sncr2\" (UniqueName: \"kubernetes.io/projected/2f1e38d2-8012-49ce-bdc4-de7d55d11239-kube-api-access-sncr2\") pod \"authorino-operator-76b478bd78-zqqmm\" (UID: \"2f1e38d2-8012-49ce-bdc4-de7d55d11239\") " pod="kuadrant-system/authorino-operator-76b478bd78-zqqmm"
Sep 18 15:42:47.277358 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:47.277277    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-zqqmm"
Sep 18 15:42:47.368143 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:47.368107    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:47.580556 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:47.580439    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:47.583966 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:47.583897    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zqqmm"]
Sep 18 15:42:47.601122 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:42:47.601090    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1e38d2_8012_49ce_bdc4_de7d55d11239.slice/crio-d14ce96c4907ef7f3cafa4f796ac7e2d439b2b93162fa31e7550ae8073fac662 WatchSource:0}: Error finding container d14ce96c4907ef7f3cafa4f796ac7e2d439b2b93162fa31e7550ae8073fac662: Status 404 returned error can't find the container with id d14ce96c4907ef7f3cafa4f796ac7e2d439b2b93162fa31e7550ae8073fac662
Sep 18 15:42:47.644307 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:47.644275    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zqqmm" event={"ID":"2f1e38d2-8012-49ce-bdc4-de7d55d11239","Type":"ContainerStarted","Data":"d14ce96c4907ef7f3cafa4f796ac7e2d439b2b93162fa31e7550ae8073fac662"}
Sep 18 15:42:48.017863 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:48.017827    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:48.404071 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:48.403989    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:49.285797 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:49.285755    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:50.282344 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:50.282304    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:50.403952 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:50.403918    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:50.659598 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:50.659509    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zqqmm" event={"ID":"2f1e38d2-8012-49ce-bdc4-de7d55d11239","Type":"ContainerStarted","Data":"1723d2be1f707a06b71d77ee8ffdfa2324764c13ae755e419f440894bd78f514"}
Sep 18 15:42:50.659762 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:50.659652    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-zqqmm"
Sep 18 15:42:50.679591 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:50.679535    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-zqqmm" podStartSLOduration=1.966435525 podStartE2EDuration="4.679520393s" totalImagesPullingTime="2.713084868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:42:50.678741891 +0000 UTC m=+493.239974043" watchObservedRunningTime="2026-09-18 15:42:50.679520393 +0000 UTC m=+493.240752532"
Sep 18 15:42:52.028340 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:52.028297    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:52.137292 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:52.137259    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:52.288662 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:52.288569    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:53.677402 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:53.677368    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lgxst"]
Sep 18 15:42:53.680732 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:53.680709    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lgxst"
Sep 18 15:42:53.683794 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:53.683770    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mmvgq\""
Sep 18 15:42:53.683965 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:53.683948    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 15:42:53.691699 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:53.691674    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lgxst"]
Sep 18 15:42:53.734759 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:53.734710    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhvdm\" (UniqueName: \"kubernetes.io/projected/a3f00e9c-d13a-4e64-acb1-f8c5754697c5-kube-api-access-hhvdm\") pod \"dns-operator-controller-manager-7b84cd7c9c-lgxst\" (UID: \"a3f00e9c-d13a-4e64-acb1-f8c5754697c5\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lgxst"
Sep 18 15:42:53.835945 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:53.835913    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhvdm\" (UniqueName: \"kubernetes.io/projected/a3f00e9c-d13a-4e64-acb1-f8c5754697c5-kube-api-access-hhvdm\") pod \"dns-operator-controller-manager-7b84cd7c9c-lgxst\" (UID: \"a3f00e9c-d13a-4e64-acb1-f8c5754697c5\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lgxst"
Sep 18 15:42:53.845101 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:53.845072    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhvdm\" (UniqueName: \"kubernetes.io/projected/a3f00e9c-d13a-4e64-acb1-f8c5754697c5-kube-api-access-hhvdm\") pod \"dns-operator-controller-manager-7b84cd7c9c-lgxst\" (UID: \"a3f00e9c-d13a-4e64-acb1-f8c5754697c5\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lgxst"
Sep 18 15:42:53.990898 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:53.990845    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lgxst"
Sep 18 15:42:54.258312 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:54.258289    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lgxst"]
Sep 18 15:42:54.260664 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:42:54.260637    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f00e9c_d13a_4e64_acb1_f8c5754697c5.slice/crio-7fdcabe94c81dc237f10fceb95194198d7e634597e6293bf358fc3efa3bef5ee WatchSource:0}: Error finding container 7fdcabe94c81dc237f10fceb95194198d7e634597e6293bf358fc3efa3bef5ee: Status 404 returned error can't find the container with id 7fdcabe94c81dc237f10fceb95194198d7e634597e6293bf358fc3efa3bef5ee
Sep 18 15:42:54.679600 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:54.679520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lgxst" event={"ID":"a3f00e9c-d13a-4e64-acb1-f8c5754697c5","Type":"ContainerStarted","Data":"7fdcabe94c81dc237f10fceb95194198d7e634597e6293bf358fc3efa3bef5ee"}
Sep 18 15:42:55.080964 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:55.080928    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:55.178033 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:55.177974    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:56.688927 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:56.688896    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lgxst" event={"ID":"a3f00e9c-d13a-4e64-acb1-f8c5754697c5","Type":"ContainerStarted","Data":"2ea2d6e8c4e0ea6fecdd8ec003a8a88651f54fc81ce206152c7aacfc12bdd081"}
Sep 18 15:42:56.689294 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:56.689030    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lgxst"
Sep 18 15:42:56.729401 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:56.729355    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lgxst" podStartSLOduration=1.905084639 podStartE2EDuration="3.729340623s" totalImagesPullingTime="1.824255984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:42:56.727152277 +0000 UTC m=+499.288384427" watchObservedRunningTime="2026-09-18 15:42:56.729340623 +0000 UTC m=+499.290572760"
Sep 18 15:42:58.287697 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:58.287664    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:58.394531 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:58.394495    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:59.027009 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:59.026975    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:59.137276 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:59.137233    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:59.260560 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:59.260524    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:42:59.358657 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:42:59.358581    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:01.666394 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:01.666354    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-zqqmm"
Sep 18 15:43:04.212102 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:04.212062    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:04.307510 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:04.307455    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:04.407997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:04.407960    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:05.150809 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:05.150739    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:05.250846 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:05.250813    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:07.694327 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:07.694296    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lgxst"
Sep 18 15:43:09.395915 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:09.395881    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"]
Sep 18 15:43:09.398971 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:09.398954    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"
Sep 18 15:43:09.402034 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:09.402010    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hq9sc\""
Sep 18 15:43:09.419019 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:09.418999    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"]
Sep 18 15:43:09.460377 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:09.460348    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc5z6\" (UniqueName: \"kubernetes.io/projected/33191069-85c1-45cf-8d14-a0540d93ea43-kube-api-access-hc5z6\") pod \"kuadrant-operator-controller-manager-5f685c496f-6hgjh\" (UID: \"33191069-85c1-45cf-8d14-a0540d93ea43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"
Sep 18 15:43:09.460511 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:09.460391    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33191069-85c1-45cf-8d14-a0540d93ea43-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-6hgjh\" (UID: \"33191069-85c1-45cf-8d14-a0540d93ea43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"
Sep 18 15:43:09.560752 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:09.560728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc5z6\" (UniqueName: \"kubernetes.io/projected/33191069-85c1-45cf-8d14-a0540d93ea43-kube-api-access-hc5z6\") pod \"kuadrant-operator-controller-manager-5f685c496f-6hgjh\" (UID: \"33191069-85c1-45cf-8d14-a0540d93ea43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"
Sep 18 15:43:09.560889 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:09.560765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33191069-85c1-45cf-8d14-a0540d93ea43-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-6hgjh\" (UID: \"33191069-85c1-45cf-8d14-a0540d93ea43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"
Sep 18 15:43:09.561083 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:09.561068    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33191069-85c1-45cf-8d14-a0540d93ea43-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-6hgjh\" (UID: \"33191069-85c1-45cf-8d14-a0540d93ea43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"
Sep 18 15:43:09.572674 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:09.572654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc5z6\" (UniqueName: \"kubernetes.io/projected/33191069-85c1-45cf-8d14-a0540d93ea43-kube-api-access-hc5z6\") pod \"kuadrant-operator-controller-manager-5f685c496f-6hgjh\" (UID: \"33191069-85c1-45cf-8d14-a0540d93ea43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"
Sep 18 15:43:09.708283 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:09.708255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"
Sep 18 15:43:09.981611 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:09.981587    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"]
Sep 18 15:43:09.983788 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:43:09.983761    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33191069_85c1_45cf_8d14_a0540d93ea43.slice/crio-f151357195e0db5633a4693d38f7d4db9af15a8d779592d35301e8335541a279 WatchSource:0}: Error finding container f151357195e0db5633a4693d38f7d4db9af15a8d779592d35301e8335541a279: Status 404 returned error can't find the container with id f151357195e0db5633a4693d38f7d4db9af15a8d779592d35301e8335541a279
Sep 18 15:43:10.043588 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.043557    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"]
Sep 18 15:43:10.047693 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.047672    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"]
Sep 18 15:43:10.071022 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.070995    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"]
Sep 18 15:43:10.075828 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.075808    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"
Sep 18 15:43:10.090928 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.090905    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"]
Sep 18 15:43:10.165837 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.165806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23eb517c-9b96-4192-8b63-58707dfc3f06-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-6dz4l\" (UID: \"23eb517c-9b96-4192-8b63-58707dfc3f06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"
Sep 18 15:43:10.166011 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.165865    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcgpn\" (UniqueName: \"kubernetes.io/projected/23eb517c-9b96-4192-8b63-58707dfc3f06-kube-api-access-bcgpn\") pod \"kuadrant-operator-controller-manager-5f685c496f-6dz4l\" (UID: \"23eb517c-9b96-4192-8b63-58707dfc3f06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"
Sep 18 15:43:10.266918 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.266807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23eb517c-9b96-4192-8b63-58707dfc3f06-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-6dz4l\" (UID: \"23eb517c-9b96-4192-8b63-58707dfc3f06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"
Sep 18 15:43:10.266918 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.266873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcgpn\" (UniqueName: \"kubernetes.io/projected/23eb517c-9b96-4192-8b63-58707dfc3f06-kube-api-access-bcgpn\") pod \"kuadrant-operator-controller-manager-5f685c496f-6dz4l\" (UID: \"23eb517c-9b96-4192-8b63-58707dfc3f06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"
Sep 18 15:43:10.267194 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.267168    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23eb517c-9b96-4192-8b63-58707dfc3f06-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-6dz4l\" (UID: \"23eb517c-9b96-4192-8b63-58707dfc3f06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"
Sep 18 15:43:10.278478 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.278459    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcgpn\" (UniqueName: \"kubernetes.io/projected/23eb517c-9b96-4192-8b63-58707dfc3f06-kube-api-access-bcgpn\") pod \"kuadrant-operator-controller-manager-5f685c496f-6dz4l\" (UID: \"23eb517c-9b96-4192-8b63-58707dfc3f06\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"
Sep 18 15:43:10.387452 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.387427    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"
Sep 18 15:43:10.708050 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.708016    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"]
Sep 18 15:43:10.712950 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:43:10.712918    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23eb517c_9b96_4192_8b63_58707dfc3f06.slice/crio-57cf895b98aa44a322e4bc8cd547e1e20d26aece5556b8cc6b2d365d8dd6763f WatchSource:0}: Error finding container 57cf895b98aa44a322e4bc8cd547e1e20d26aece5556b8cc6b2d365d8dd6763f: Status 404 returned error can't find the container with id 57cf895b98aa44a322e4bc8cd547e1e20d26aece5556b8cc6b2d365d8dd6763f
Sep 18 15:43:10.752129 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.752094    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l" event={"ID":"23eb517c-9b96-4192-8b63-58707dfc3f06","Type":"ContainerStarted","Data":"57cf895b98aa44a322e4bc8cd547e1e20d26aece5556b8cc6b2d365d8dd6763f"}
Sep 18 15:43:10.888444 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:10.888410    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:11.007531 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:11.007444    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:14.774178 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:14.774139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l" event={"ID":"23eb517c-9b96-4192-8b63-58707dfc3f06","Type":"ContainerStarted","Data":"832e64612745b1af77c038648c5d8e6d70fa5f4e1f3b4335ed3f2bb0f93cc756"}
Sep 18 15:43:14.774613 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:14.774212    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"
Sep 18 15:43:14.775787 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:14.775756    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh" podUID="33191069-85c1-45cf-8d14-a0540d93ea43" containerName="manager" containerID="cri-o://a2bd06fecd0802a70a8ebc0611e941f17385b75af60b8f47cab063fa19697bae" gracePeriod=2
Sep 18 15:43:14.800808 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:14.800766    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l" podStartSLOduration=1.7527162509999998 podStartE2EDuration="4.800750791s" totalImagesPullingTime="3.04803454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:43:14.798438591 +0000 UTC m=+517.359670733" watchObservedRunningTime="2026-09-18 15:43:14.800750791 +0000 UTC m=+517.361982931"
Sep 18 15:43:15.008998 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.008975    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"
Sep 18 15:43:15.011704 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:43:15.011678    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-6hgjh\" is forbidden: User \"system:node:ip-10-0-130-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-223.ec2.internal' and this object" podUID="33191069-85c1-45cf-8d14-a0540d93ea43" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"
Sep 18 15:43:15.065633 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.065546    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:15.109768 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.109736    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc5z6\" (UniqueName: \"kubernetes.io/projected/33191069-85c1-45cf-8d14-a0540d93ea43-kube-api-access-hc5z6\") pod \"33191069-85c1-45cf-8d14-a0540d93ea43\" (UID: \"33191069-85c1-45cf-8d14-a0540d93ea43\") "
Sep 18 15:43:15.109940 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.109832    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33191069-85c1-45cf-8d14-a0540d93ea43-extensions-socket-volume\") pod \"33191069-85c1-45cf-8d14-a0540d93ea43\" (UID: \"33191069-85c1-45cf-8d14-a0540d93ea43\") "
Sep 18 15:43:15.110089 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.110065    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33191069-85c1-45cf-8d14-a0540d93ea43-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "33191069-85c1-45cf-8d14-a0540d93ea43" (UID: "33191069-85c1-45cf-8d14-a0540d93ea43"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:43:15.112022 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.112003    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33191069-85c1-45cf-8d14-a0540d93ea43-kube-api-access-hc5z6" (OuterVolumeSpecName: "kube-api-access-hc5z6") pod "33191069-85c1-45cf-8d14-a0540d93ea43" (UID: "33191069-85c1-45cf-8d14-a0540d93ea43"). InnerVolumeSpecName "kube-api-access-hc5z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:43:15.168687 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.168654    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:15.211090 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.211060    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc5z6\" (UniqueName: \"kubernetes.io/projected/33191069-85c1-45cf-8d14-a0540d93ea43-kube-api-access-hc5z6\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:43:15.211090 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.211084    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33191069-85c1-45cf-8d14-a0540d93ea43-extensions-socket-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:43:15.743350 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.743314    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33191069-85c1-45cf-8d14-a0540d93ea43" path="/var/lib/kubelet/pods/33191069-85c1-45cf-8d14-a0540d93ea43/volumes"
Sep 18 15:43:15.781041 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.781011    2559 generic.go:356] "Generic (PLEG): container finished" podID="33191069-85c1-45cf-8d14-a0540d93ea43" containerID="a2bd06fecd0802a70a8ebc0611e941f17385b75af60b8f47cab063fa19697bae" exitCode=2
Sep 18 15:43:15.781409 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.781056    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"
Sep 18 15:43:15.781409 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.781127    2559 scope.go:117] "RemoveContainer" containerID="a2bd06fecd0802a70a8ebc0611e941f17385b75af60b8f47cab063fa19697bae"
Sep 18 15:43:15.785909 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:43:15.785883    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-6hgjh\" is forbidden: User \"system:node:ip-10-0-130-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-223.ec2.internal' and this object" podUID="33191069-85c1-45cf-8d14-a0540d93ea43" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6hgjh"
Sep 18 15:43:15.789299 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.789278    2559 scope.go:117] "RemoveContainer" containerID="a2bd06fecd0802a70a8ebc0611e941f17385b75af60b8f47cab063fa19697bae"
Sep 18 15:43:15.789538 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:43:15.789517    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bd06fecd0802a70a8ebc0611e941f17385b75af60b8f47cab063fa19697bae\": container with ID starting with a2bd06fecd0802a70a8ebc0611e941f17385b75af60b8f47cab063fa19697bae not found: ID does not exist" containerID="a2bd06fecd0802a70a8ebc0611e941f17385b75af60b8f47cab063fa19697bae"
Sep 18 15:43:15.789603 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:15.789550    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bd06fecd0802a70a8ebc0611e941f17385b75af60b8f47cab063fa19697bae"} err="failed to get container status \"a2bd06fecd0802a70a8ebc0611e941f17385b75af60b8f47cab063fa19697bae\": rpc error: code = NotFound desc = could not find container \"a2bd06fecd0802a70a8ebc0611e941f17385b75af60b8f47cab063fa19697bae\": container with ID starting with a2bd06fecd0802a70a8ebc0611e941f17385b75af60b8f47cab063fa19697bae not found: ID does not exist"
Sep 18 15:43:18.857011 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:18.856976    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:18.916777 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:18.916734    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:19.055696 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:19.055665    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:19.157936 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:19.157832    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:24.450246 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:24.450153    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:24.515712 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:24.515680    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:24.654892 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:24.654848    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:24.714116 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:24.714031    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:24.859327 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:24.859294    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:24.932215 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:24.932180    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:25.783973 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:25.783941    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"
Sep 18 15:43:30.818989 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:30.818956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:30.884599 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:30.884562    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:33.189308 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:33.189265    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:38.960888 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:38.960841    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"]
Sep 18 15:43:38.961315 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:38.961054    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l" podUID="23eb517c-9b96-4192-8b63-58707dfc3f06" containerName="manager" containerID="cri-o://832e64612745b1af77c038648c5d8e6d70fa5f4e1f3b4335ed3f2bb0f93cc756" gracePeriod=10
Sep 18 15:43:39.200654 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.200627    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"
Sep 18 15:43:39.316010 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.315922    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23eb517c-9b96-4192-8b63-58707dfc3f06-extensions-socket-volume\") pod \"23eb517c-9b96-4192-8b63-58707dfc3f06\" (UID: \"23eb517c-9b96-4192-8b63-58707dfc3f06\") "
Sep 18 15:43:39.316010 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.315985    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcgpn\" (UniqueName: \"kubernetes.io/projected/23eb517c-9b96-4192-8b63-58707dfc3f06-kube-api-access-bcgpn\") pod \"23eb517c-9b96-4192-8b63-58707dfc3f06\" (UID: \"23eb517c-9b96-4192-8b63-58707dfc3f06\") "
Sep 18 15:43:39.316466 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.316432    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23eb517c-9b96-4192-8b63-58707dfc3f06-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "23eb517c-9b96-4192-8b63-58707dfc3f06" (UID: "23eb517c-9b96-4192-8b63-58707dfc3f06"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:43:39.318032 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.317996    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23eb517c-9b96-4192-8b63-58707dfc3f06-kube-api-access-bcgpn" (OuterVolumeSpecName: "kube-api-access-bcgpn") pod "23eb517c-9b96-4192-8b63-58707dfc3f06" (UID: "23eb517c-9b96-4192-8b63-58707dfc3f06"). InnerVolumeSpecName "kube-api-access-bcgpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:43:39.417296 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.417267    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23eb517c-9b96-4192-8b63-58707dfc3f06-extensions-socket-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:43:39.417296 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.417293    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcgpn\" (UniqueName: \"kubernetes.io/projected/23eb517c-9b96-4192-8b63-58707dfc3f06-kube-api-access-bcgpn\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:43:39.886362 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.886325    2559 generic.go:356] "Generic (PLEG): container finished" podID="23eb517c-9b96-4192-8b63-58707dfc3f06" containerID="832e64612745b1af77c038648c5d8e6d70fa5f4e1f3b4335ed3f2bb0f93cc756" exitCode=0
Sep 18 15:43:39.886362 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.886364    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"
Sep 18 15:43:39.886562 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.886413    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l" event={"ID":"23eb517c-9b96-4192-8b63-58707dfc3f06","Type":"ContainerDied","Data":"832e64612745b1af77c038648c5d8e6d70fa5f4e1f3b4335ed3f2bb0f93cc756"}
Sep 18 15:43:39.886562 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.886459    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l" event={"ID":"23eb517c-9b96-4192-8b63-58707dfc3f06","Type":"ContainerDied","Data":"57cf895b98aa44a322e4bc8cd547e1e20d26aece5556b8cc6b2d365d8dd6763f"}
Sep 18 15:43:39.886562 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.886486    2559 scope.go:117] "RemoveContainer" containerID="832e64612745b1af77c038648c5d8e6d70fa5f4e1f3b4335ed3f2bb0f93cc756"
Sep 18 15:43:39.894527 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.894512    2559 scope.go:117] "RemoveContainer" containerID="832e64612745b1af77c038648c5d8e6d70fa5f4e1f3b4335ed3f2bb0f93cc756"
Sep 18 15:43:39.894765 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:43:39.894746    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832e64612745b1af77c038648c5d8e6d70fa5f4e1f3b4335ed3f2bb0f93cc756\": container with ID starting with 832e64612745b1af77c038648c5d8e6d70fa5f4e1f3b4335ed3f2bb0f93cc756 not found: ID does not exist" containerID="832e64612745b1af77c038648c5d8e6d70fa5f4e1f3b4335ed3f2bb0f93cc756"
Sep 18 15:43:39.894817 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.894773    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832e64612745b1af77c038648c5d8e6d70fa5f4e1f3b4335ed3f2bb0f93cc756"} err="failed to get container status \"832e64612745b1af77c038648c5d8e6d70fa5f4e1f3b4335ed3f2bb0f93cc756\": rpc error: code = NotFound desc = could not find container \"832e64612745b1af77c038648c5d8e6d70fa5f4e1f3b4335ed3f2bb0f93cc756\": container with ID starting with 832e64612745b1af77c038648c5d8e6d70fa5f4e1f3b4335ed3f2bb0f93cc756 not found: ID does not exist"
Sep 18 15:43:39.906936 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.906912    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"]
Sep 18 15:43:39.913310 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:39.913291    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-6dz4l"]
Sep 18 15:43:41.747994 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:41.747964    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23eb517c-9b96-4192-8b63-58707dfc3f06" path="/var/lib/kubelet/pods/23eb517c-9b96-4192-8b63-58707dfc3f06/volumes"
Sep 18 15:43:42.194971 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:42.194943    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:42.415333 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:42.415295    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:57.870825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:57.870785    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:57.966334 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:57.966295    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:58.063931 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.063869    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:58.270995 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.270960    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:43:58.271271 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.271259    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33191069-85c1-45cf-8d14-a0540d93ea43" containerName="manager"
Sep 18 15:43:58.271325 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.271273    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="33191069-85c1-45cf-8d14-a0540d93ea43" containerName="manager"
Sep 18 15:43:58.271325 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.271295    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23eb517c-9b96-4192-8b63-58707dfc3f06" containerName="manager"
Sep 18 15:43:58.271325 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.271301    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="23eb517c-9b96-4192-8b63-58707dfc3f06" containerName="manager"
Sep 18 15:43:58.271445 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.271362    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="33191069-85c1-45cf-8d14-a0540d93ea43" containerName="manager"
Sep 18 15:43:58.271445 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.271372    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="23eb517c-9b96-4192-8b63-58707dfc3f06" containerName="manager"
Sep 18 15:43:58.274186 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.274171    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"
Sep 18 15:43:58.277016 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.276993    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 15:43:58.277131 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.276993    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f6brr\""
Sep 18 15:43:58.283194 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.283174    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:43:58.299278 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.299250    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:43:58.363124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.363083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8tp\" (UniqueName: \"kubernetes.io/projected/b5947521-edec-444b-abe3-33c010639ce1-kube-api-access-7v8tp\") pod \"limitador-limitador-84994f7ffc-rlx5z\" (UID: \"b5947521-edec-444b-abe3-33c010639ce1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"
Sep 18 15:43:58.363292 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.363150    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b5947521-edec-444b-abe3-33c010639ce1-config-file\") pod \"limitador-limitador-84994f7ffc-rlx5z\" (UID: \"b5947521-edec-444b-abe3-33c010639ce1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"
Sep 18 15:43:58.464254 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.464216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b5947521-edec-444b-abe3-33c010639ce1-config-file\") pod \"limitador-limitador-84994f7ffc-rlx5z\" (UID: \"b5947521-edec-444b-abe3-33c010639ce1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"
Sep 18 15:43:58.464433 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.464281    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8tp\" (UniqueName: \"kubernetes.io/projected/b5947521-edec-444b-abe3-33c010639ce1-kube-api-access-7v8tp\") pod \"limitador-limitador-84994f7ffc-rlx5z\" (UID: \"b5947521-edec-444b-abe3-33c010639ce1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"
Sep 18 15:43:58.464885 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.464837    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b5947521-edec-444b-abe3-33c010639ce1-config-file\") pod \"limitador-limitador-84994f7ffc-rlx5z\" (UID: \"b5947521-edec-444b-abe3-33c010639ce1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"
Sep 18 15:43:58.473907 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.473886    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8tp\" (UniqueName: \"kubernetes.io/projected/b5947521-edec-444b-abe3-33c010639ce1-kube-api-access-7v8tp\") pod \"limitador-limitador-84994f7ffc-rlx5z\" (UID: \"b5947521-edec-444b-abe3-33c010639ce1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"
Sep 18 15:43:58.584837 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.584752    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"
Sep 18 15:43:58.712962 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.712925    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:58.795477 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.795448    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:43:58.852377 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.852354    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:43:58.854779 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:43:58.854751    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5947521_edec_444b_abe3_33c010639ce1.slice/crio-91218d2af3451fb18eab1b9d299b93242ff580f2ac1b03b22ec869a4a0e62e02 WatchSource:0}: Error finding container 91218d2af3451fb18eab1b9d299b93242ff580f2ac1b03b22ec869a4a0e62e02: Status 404 returned error can't find the container with id 91218d2af3451fb18eab1b9d299b93242ff580f2ac1b03b22ec869a4a0e62e02
Sep 18 15:43:58.969808 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:58.969771    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-rlx5z" event={"ID":"b5947521-edec-444b-abe3-33c010639ce1","Type":"ContainerStarted","Data":"91218d2af3451fb18eab1b9d299b93242ff580f2ac1b03b22ec869a4a0e62e02"}
Sep 18 15:43:59.976216 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:59.976185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-rlx5z" event={"ID":"b5947521-edec-444b-abe3-33c010639ce1","Type":"ContainerStarted","Data":"2809002012e704695a623f471aaf0666eff279816d6f3b234d1d979a5b530087"}
Sep 18 15:43:59.976586 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:59.976322    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"
Sep 18 15:43:59.993999 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:43:59.993915    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-rlx5z" podStartSLOduration=1.11701895 podStartE2EDuration="1.993900744s" totalImagesPullingTime="876.881794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:43:59.992690568 +0000 UTC m=+562.553922706" watchObservedRunningTime="2026-09-18 15:43:59.993900744 +0000 UTC m=+562.555132883"
Sep 18 15:44:09.257351 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:09.257312    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:44:09.319017 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:09.318991    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:44:10.980700 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:10.980671    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"
Sep 18 15:44:15.475386 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:15.475347    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:44:15.578766 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:15.578727    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:44:15.926777 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:15.926686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:44:28.491912 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:28.491872    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:44:28.740706 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:28.740663    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:44:35.923822 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:35.923784    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:44:36.018348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:36.018314    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:44:36.088730 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:36.088696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:44:36.312260 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:36.312231    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:44:39.386552 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:39.386518    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:44:39.516331 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:44:39.516299    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:11.993048 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:11.992995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:13.165495 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:13.165456    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:13.591225 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:13.591193    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:13.692753 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:13.692723    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:13.799485 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:13.799450    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:14.105316 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.105285    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:14.205770 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.205734    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:14.269374 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.269340    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7r97d"]
Sep 18 15:45:14.272664 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.272644    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d"
Sep 18 15:45:14.276299 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.276274    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 15:45:14.276414 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.276299    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gnptn\""
Sep 18 15:45:14.277134 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.277118    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 15:45:14.278703 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.278678    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 18 15:45:14.299036 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.299006    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7r97d"]
Sep 18 15:45:14.316663 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.316630    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:14.348742 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.348706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmrfp\" (UniqueName: \"kubernetes.io/projected/d51f861a-aea0-407c-bf85-09b43d4f8266-kube-api-access-zmrfp\") pod \"payload-processing-5b54b7fbb7-7r97d\" (UID: \"d51f861a-aea0-407c-bf85-09b43d4f8266\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d"
Sep 18 15:45:14.348908 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.348787    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51f861a-aea0-407c-bf85-09b43d4f8266-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7r97d\" (UID: \"d51f861a-aea0-407c-bf85-09b43d4f8266\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d"
Sep 18 15:45:14.366879 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.366796    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:45:14.449452 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.449415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51f861a-aea0-407c-bf85-09b43d4f8266-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7r97d\" (UID: \"d51f861a-aea0-407c-bf85-09b43d4f8266\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d"
Sep 18 15:45:14.449613 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.449494    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmrfp\" (UniqueName: \"kubernetes.io/projected/d51f861a-aea0-407c-bf85-09b43d4f8266-kube-api-access-zmrfp\") pod \"payload-processing-5b54b7fbb7-7r97d\" (UID: \"d51f861a-aea0-407c-bf85-09b43d4f8266\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d"
Sep 18 15:45:14.450150 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.450122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51f861a-aea0-407c-bf85-09b43d4f8266-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7r97d\" (UID: \"d51f861a-aea0-407c-bf85-09b43d4f8266\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d"
Sep 18 15:45:14.460522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.460497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmrfp\" (UniqueName: \"kubernetes.io/projected/d51f861a-aea0-407c-bf85-09b43d4f8266-kube-api-access-zmrfp\") pod \"payload-processing-5b54b7fbb7-7r97d\" (UID: \"d51f861a-aea0-407c-bf85-09b43d4f8266\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d"
Sep 18 15:45:14.583122 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.583088    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d"
Sep 18 15:45:14.856021 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:14.855987    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7r97d"]
Sep 18 15:45:14.859109 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:45:14.859078    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51f861a_aea0_407c_bf85_09b43d4f8266.slice/crio-f63f3f65921172b7caaab320c3338c137aadc67d705a7eec7cd17eb554f0e875 WatchSource:0}: Error finding container f63f3f65921172b7caaab320c3338c137aadc67d705a7eec7cd17eb554f0e875: Status 404 returned error can't find the container with id f63f3f65921172b7caaab320c3338c137aadc67d705a7eec7cd17eb554f0e875
Sep 18 15:45:15.289146 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:15.289112    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d" event={"ID":"d51f861a-aea0-407c-bf85-09b43d4f8266","Type":"ContainerStarted","Data":"f63f3f65921172b7caaab320c3338c137aadc67d705a7eec7cd17eb554f0e875"}
Sep 18 15:45:18.308006 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:18.307972    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d" event={"ID":"d51f861a-aea0-407c-bf85-09b43d4f8266","Type":"ContainerStarted","Data":"a9df568da0356cf52bba45b2866d7c48a31a5e776b2de0af023cae2e14ae8c98"}
Sep 18 15:45:18.308368 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:18.308129    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d"
Sep 18 15:45:18.325057 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:18.324907    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d" podStartSLOduration=1.3087714799999999 podStartE2EDuration="4.324887993s" totalImagesPullingTime="3.016116513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:45:18.324833723 +0000 UTC m=+640.886065853" watchObservedRunningTime="2026-09-18 15:45:18.324887993 +0000 UTC m=+640.886120149"
Sep 18 15:45:25.586413 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:25.586372    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:25.720995 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:25.720954    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:29.314949 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:29.314920    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d"
Sep 18 15:45:39.648668 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:39.648638    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:40.168941 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:40.168906    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:45:44.560282 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:44.560241    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:45.269637 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:45.269598    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:45:50.249518 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:50.249482    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:51.169628 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:51.169594    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:45:58.918365 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:58.918332    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"]
Sep 18 15:45:58.921090 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:58.921067    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:58.925159 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:58.924950    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 15:45:58.925159 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:58.924975    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 15:45:58.925159 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:58.924994    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7lm9q\""
Sep 18 15:45:58.925159 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:58.924995    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 18 15:45:58.933734 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:58.933714    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"]
Sep 18 15:45:58.954289 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:58.954265    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:45:59.016613 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.016587    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.016757 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.016623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwvc\" (UniqueName: \"kubernetes.io/projected/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-kube-api-access-5hwvc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.016757 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.016655    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.016757 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.016684    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.016757 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.016711    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.016933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.016758    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.117321 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.117287    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.117321 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.117326    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwvc\" (UniqueName: \"kubernetes.io/projected/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-kube-api-access-5hwvc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.117511 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.117345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.117511 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.117368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.117511 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.117496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.117630 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.117534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.117825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.117790    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.117825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.117809    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.118009 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.117834    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.119509 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.119486    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.119899 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.119880    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.129929 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.126990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwvc\" (UniqueName: \"kubernetes.io/projected/aba4efba-0f1d-408c-b6f5-5e92a418cc6f-kube-api-access-5hwvc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hlsmb\" (UID: \"aba4efba-0f1d-408c-b6f5-5e92a418cc6f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.231910 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.231875    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:45:59.498803 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.498732    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"]
Sep 18 15:45:59.501456 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:45:59.501430    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba4efba_0f1d_408c_b6f5_5e92a418cc6f.slice/crio-10b6b641b01d68b1e67f538903c5727703c350485fe47298575ac0260cf8eb26 WatchSource:0}: Error finding container 10b6b641b01d68b1e67f538903c5727703c350485fe47298575ac0260cf8eb26: Status 404 returned error can't find the container with id 10b6b641b01d68b1e67f538903c5727703c350485fe47298575ac0260cf8eb26
Sep 18 15:45:59.502663 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:45:59.502646    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 15:46:00.272957 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:00.272920    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:46:00.484092 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:00.484057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb" event={"ID":"aba4efba-0f1d-408c-b6f5-5e92a418cc6f","Type":"ContainerStarted","Data":"10b6b641b01d68b1e67f538903c5727703c350485fe47298575ac0260cf8eb26"}
Sep 18 15:46:01.489871 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:01.489822    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb" event={"ID":"aba4efba-0f1d-408c-b6f5-5e92a418cc6f","Type":"ContainerStarted","Data":"42f912b9b93660b3831f37e2b45cceb7971fe3bd37c2e2de27935db90b0ccea7"}
Sep 18 15:46:01.511663 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:01.511617    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb" podStartSLOduration=2.423371791 podStartE2EDuration="3.51160275s" totalImagesPullingTime="1.088230959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:46:01.510487364 +0000 UTC m=+684.071719503" watchObservedRunningTime="2026-09-18 15:46:01.51160275 +0000 UTC m=+684.072834888"
Sep 18 15:46:09.232949 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:09.232917    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:46:09.233306 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:09.232974    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:46:09.244528 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:09.244506    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:46:09.533490 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:09.533417    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hlsmb"
Sep 18 15:46:14.158755 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:14.158724    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:46:15.408167 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:15.408130    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:46:15.466321 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:15.466292    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:46:15.678943 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:15.678868    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:46:23.451657 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:23.451578    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:46:25.271892 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:25.271846    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:46:31.115567 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:31.115534    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:46:33.475599 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:46:33.475565    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:47:08.487710 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:08.487675    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:47:08.668491 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:08.668459    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7r97d"]
Sep 18 15:47:08.668674 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:08.668652    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d" podUID="d51f861a-aea0-407c-bf85-09b43d4f8266" containerName="payload-processing" containerID="cri-o://a9df568da0356cf52bba45b2866d7c48a31a5e776b2de0af023cae2e14ae8c98" gracePeriod=30
Sep 18 15:47:08.883156 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:08.883120    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:47:08.905693 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:08.905672    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d"
Sep 18 15:47:09.026628 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.026594    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:47:09.056766 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.056709    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmrfp\" (UniqueName: \"kubernetes.io/projected/d51f861a-aea0-407c-bf85-09b43d4f8266-kube-api-access-zmrfp\") pod \"d51f861a-aea0-407c-bf85-09b43d4f8266\" (UID: \"d51f861a-aea0-407c-bf85-09b43d4f8266\") "
Sep 18 15:47:09.056766 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.056753    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51f861a-aea0-407c-bf85-09b43d4f8266-plugins-config-volume\") pod \"d51f861a-aea0-407c-bf85-09b43d4f8266\" (UID: \"d51f861a-aea0-407c-bf85-09b43d4f8266\") "
Sep 18 15:47:09.057167 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.057148    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d51f861a-aea0-407c-bf85-09b43d4f8266-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d51f861a-aea0-407c-bf85-09b43d4f8266" (UID: "d51f861a-aea0-407c-bf85-09b43d4f8266"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:47:09.058804 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.058777    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d51f861a-aea0-407c-bf85-09b43d4f8266-kube-api-access-zmrfp" (OuterVolumeSpecName: "kube-api-access-zmrfp") pod "d51f861a-aea0-407c-bf85-09b43d4f8266" (UID: "d51f861a-aea0-407c-bf85-09b43d4f8266"). InnerVolumeSpecName "kube-api-access-zmrfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:47:09.157658 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.157636    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmrfp\" (UniqueName: \"kubernetes.io/projected/d51f861a-aea0-407c-bf85-09b43d4f8266-kube-api-access-zmrfp\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:47:09.157658 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.157657    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51f861a-aea0-407c-bf85-09b43d4f8266-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:47:09.769191 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.769161    2559 generic.go:356] "Generic (PLEG): container finished" podID="d51f861a-aea0-407c-bf85-09b43d4f8266" containerID="a9df568da0356cf52bba45b2866d7c48a31a5e776b2de0af023cae2e14ae8c98" exitCode=0
Sep 18 15:47:09.769544 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.769197    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d"
Sep 18 15:47:09.769544 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.769235    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d" event={"ID":"d51f861a-aea0-407c-bf85-09b43d4f8266","Type":"ContainerDied","Data":"a9df568da0356cf52bba45b2866d7c48a31a5e776b2de0af023cae2e14ae8c98"}
Sep 18 15:47:09.769544 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.769279    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7r97d" event={"ID":"d51f861a-aea0-407c-bf85-09b43d4f8266","Type":"ContainerDied","Data":"f63f3f65921172b7caaab320c3338c137aadc67d705a7eec7cd17eb554f0e875"}
Sep 18 15:47:09.769544 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.769300    2559 scope.go:117] "RemoveContainer" containerID="a9df568da0356cf52bba45b2866d7c48a31a5e776b2de0af023cae2e14ae8c98"
Sep 18 15:47:09.777982 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.777954    2559 scope.go:117] "RemoveContainer" containerID="a9df568da0356cf52bba45b2866d7c48a31a5e776b2de0af023cae2e14ae8c98"
Sep 18 15:47:09.778310 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:47:09.778281    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9df568da0356cf52bba45b2866d7c48a31a5e776b2de0af023cae2e14ae8c98\": container with ID starting with a9df568da0356cf52bba45b2866d7c48a31a5e776b2de0af023cae2e14ae8c98 not found: ID does not exist" containerID="a9df568da0356cf52bba45b2866d7c48a31a5e776b2de0af023cae2e14ae8c98"
Sep 18 15:47:09.778410 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.778319    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9df568da0356cf52bba45b2866d7c48a31a5e776b2de0af023cae2e14ae8c98"} err="failed to get container status \"a9df568da0356cf52bba45b2866d7c48a31a5e776b2de0af023cae2e14ae8c98\": rpc error: code = NotFound desc = could not find container \"a9df568da0356cf52bba45b2866d7c48a31a5e776b2de0af023cae2e14ae8c98\": container with ID starting with a9df568da0356cf52bba45b2866d7c48a31a5e776b2de0af023cae2e14ae8c98 not found: ID does not exist"
Sep 18 15:47:09.791254 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.791232    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7r97d"]
Sep 18 15:47:09.797957 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:09.797936    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7r97d"]
Sep 18 15:47:11.743743 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:11.743711    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d51f861a-aea0-407c-bf85-09b43d4f8266" path="/var/lib/kubelet/pods/d51f861a-aea0-407c-bf85-09b43d4f8266/volumes"
Sep 18 15:47:22.232513 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.232471    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-586bd8bd6-8rr92"]
Sep 18 15:47:22.232985 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.232956    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d51f861a-aea0-407c-bf85-09b43d4f8266" containerName="payload-processing"
Sep 18 15:47:22.232985 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.232988    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d51f861a-aea0-407c-bf85-09b43d4f8266" containerName="payload-processing"
Sep 18 15:47:22.233122 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.233090    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d51f861a-aea0-407c-bf85-09b43d4f8266" containerName="payload-processing"
Sep 18 15:47:22.235905 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.235885    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92"
Sep 18 15:47:22.239071 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.239052    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Sep 18 15:47:22.239155 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.239058    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-4f8zv\""
Sep 18 15:47:22.244168 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.244144    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-586bd8bd6-8rr92"]
Sep 18 15:47:22.262269 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.262240    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfsr4\" (UniqueName: \"kubernetes.io/projected/0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2-kube-api-access-gfsr4\") pod \"ai-gateway-controller-586bd8bd6-8rr92\" (UID: \"0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2\") " pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92"
Sep 18 15:47:22.363501 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.363474    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfsr4\" (UniqueName: \"kubernetes.io/projected/0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2-kube-api-access-gfsr4\") pod \"ai-gateway-controller-586bd8bd6-8rr92\" (UID: \"0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2\") " pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92"
Sep 18 15:47:22.372675 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.372656    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfsr4\" (UniqueName: \"kubernetes.io/projected/0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2-kube-api-access-gfsr4\") pod \"ai-gateway-controller-586bd8bd6-8rr92\" (UID: \"0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2\") " pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92"
Sep 18 15:47:22.545446 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.545365    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92"
Sep 18 15:47:22.811187 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.811163    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-586bd8bd6-8rr92"]
Sep 18 15:47:22.813191 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:47:22.813158    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf9ebc3_a24f_424b_8b17_6f867ff0c0e2.slice/crio-01e565e73185a80de99d7942847031100fa9d445cb8e9c12ad3d8bd5d018cbbc WatchSource:0}: Error finding container 01e565e73185a80de99d7942847031100fa9d445cb8e9c12ad3d8bd5d018cbbc: Status 404 returned error can't find the container with id 01e565e73185a80de99d7942847031100fa9d445cb8e9c12ad3d8bd5d018cbbc
Sep 18 15:47:22.824906 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:22.824876    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92" event={"ID":"0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2","Type":"ContainerStarted","Data":"01e565e73185a80de99d7942847031100fa9d445cb8e9c12ad3d8bd5d018cbbc"}
Sep 18 15:47:25.841960 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:25.841923    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92" event={"ID":"0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2","Type":"ContainerStarted","Data":"8677ab41e0130070e26543aff05fbfcf319c628744c0c1cbd263daec605f0ecf"}
Sep 18 15:47:25.842348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:25.842113    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92"
Sep 18 15:47:25.859383 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:25.859337    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92" podStartSLOduration=1.6558898370000001 podStartE2EDuration="3.859325483s" totalImagesPullingTime="2.203435646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:47:25.858977917 +0000 UTC m=+768.420210068" watchObservedRunningTime="2026-09-18 15:47:25.859325483 +0000 UTC m=+768.420557622"
Sep 18 15:47:29.514205 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.514174    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:47:29.553104 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.553075    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v"]
Sep 18 15:47:29.555363 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.555346    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v"
Sep 18 15:47:29.557941 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.557920    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 18 15:47:29.558053 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.557920    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 15:47:29.558053 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.557987    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 15:47:29.565977 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.565952    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-mx9mf"]
Sep 18 15:47:29.568008 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.567990    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-85fd65c94c-mx9mf"
Sep 18 15:47:29.570947 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.570927    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9q6mp\""
Sep 18 15:47:29.573183 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.573163    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v"]
Sep 18 15:47:29.582306 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.582283    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-mx9mf"]
Sep 18 15:47:29.618216 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.618190    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:47:29.624337 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.624314    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvlp9\" (UniqueName: \"kubernetes.io/projected/619761f1-b0a4-4b32-978d-5c3702133c7d-kube-api-access-qvlp9\") pod \"payload-processing-85fd65c94c-mx9mf\" (UID: \"619761f1-b0a4-4b32-978d-5c3702133c7d\") " pod="openshift-ingress/payload-processing-85fd65c94c-mx9mf"
Sep 18 15:47:29.624431 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.624368    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619761f1-b0a4-4b32-978d-5c3702133c7d-plugins-config-volume\") pod \"payload-processing-85fd65c94c-mx9mf\" (UID: \"619761f1-b0a4-4b32-978d-5c3702133c7d\") " pod="openshift-ingress/payload-processing-85fd65c94c-mx9mf"
Sep 18 15:47:29.624502 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.624444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30914206-a17b-456a-89cc-decc238d701f-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-z8d9v\" (UID: \"30914206-a17b-456a-89cc-decc238d701f\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v"
Sep 18 15:47:29.624557 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.624499    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf967\" (UniqueName: \"kubernetes.io/projected/30914206-a17b-456a-89cc-decc238d701f-kube-api-access-wf967\") pod \"payload-pre-processing-76f657cdc4-z8d9v\" (UID: \"30914206-a17b-456a-89cc-decc238d701f\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v"
Sep 18 15:47:29.725368 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.725335    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf967\" (UniqueName: \"kubernetes.io/projected/30914206-a17b-456a-89cc-decc238d701f-kube-api-access-wf967\") pod \"payload-pre-processing-76f657cdc4-z8d9v\" (UID: \"30914206-a17b-456a-89cc-decc238d701f\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v"
Sep 18 15:47:29.725368 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.725371    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvlp9\" (UniqueName: \"kubernetes.io/projected/619761f1-b0a4-4b32-978d-5c3702133c7d-kube-api-access-qvlp9\") pod \"payload-processing-85fd65c94c-mx9mf\" (UID: \"619761f1-b0a4-4b32-978d-5c3702133c7d\") " pod="openshift-ingress/payload-processing-85fd65c94c-mx9mf"
Sep 18 15:47:29.725545 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.725402    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619761f1-b0a4-4b32-978d-5c3702133c7d-plugins-config-volume\") pod \"payload-processing-85fd65c94c-mx9mf\" (UID: \"619761f1-b0a4-4b32-978d-5c3702133c7d\") " pod="openshift-ingress/payload-processing-85fd65c94c-mx9mf"
Sep 18 15:47:29.725545 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.725429    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30914206-a17b-456a-89cc-decc238d701f-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-z8d9v\" (UID: \"30914206-a17b-456a-89cc-decc238d701f\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v"
Sep 18 15:47:29.731119 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.726914    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30914206-a17b-456a-89cc-decc238d701f-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-z8d9v\" (UID: \"30914206-a17b-456a-89cc-decc238d701f\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v"
Sep 18 15:47:29.731119 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.726924    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619761f1-b0a4-4b32-978d-5c3702133c7d-plugins-config-volume\") pod \"payload-processing-85fd65c94c-mx9mf\" (UID: \"619761f1-b0a4-4b32-978d-5c3702133c7d\") " pod="openshift-ingress/payload-processing-85fd65c94c-mx9mf"
Sep 18 15:47:29.734946 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.734925    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvlp9\" (UniqueName: \"kubernetes.io/projected/619761f1-b0a4-4b32-978d-5c3702133c7d-kube-api-access-qvlp9\") pod \"payload-processing-85fd65c94c-mx9mf\" (UID: \"619761f1-b0a4-4b32-978d-5c3702133c7d\") " pod="openshift-ingress/payload-processing-85fd65c94c-mx9mf"
Sep 18 15:47:29.735042 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.734998    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf967\" (UniqueName: \"kubernetes.io/projected/30914206-a17b-456a-89cc-decc238d701f-kube-api-access-wf967\") pod \"payload-pre-processing-76f657cdc4-z8d9v\" (UID: \"30914206-a17b-456a-89cc-decc238d701f\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v"
Sep 18 15:47:29.864527 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.864446    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v"
Sep 18 15:47:29.878613 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:29.878588    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-85fd65c94c-mx9mf"
Sep 18 15:47:30.138723 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:30.138692    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v"]
Sep 18 15:47:30.140691 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:47:30.140664    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30914206_a17b_456a_89cc_decc238d701f.slice/crio-f2521d813fe32b1b1fe1643a77a5623b9fb9b70e0bda7b0144d56633c425a186 WatchSource:0}: Error finding container f2521d813fe32b1b1fe1643a77a5623b9fb9b70e0bda7b0144d56633c425a186: Status 404 returned error can't find the container with id f2521d813fe32b1b1fe1643a77a5623b9fb9b70e0bda7b0144d56633c425a186
Sep 18 15:47:30.155559 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:30.155533    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-mx9mf"]
Sep 18 15:47:30.157197 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:47:30.157170    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619761f1_b0a4_4b32_978d_5c3702133c7d.slice/crio-160937ee4fec9f1505646fa841d05488c201f823dec7fa41b39e4f84d4b21815 WatchSource:0}: Error finding container 160937ee4fec9f1505646fa841d05488c201f823dec7fa41b39e4f84d4b21815: Status 404 returned error can't find the container with id 160937ee4fec9f1505646fa841d05488c201f823dec7fa41b39e4f84d4b21815
Sep 18 15:47:30.862930 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:30.862895    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-85fd65c94c-mx9mf" event={"ID":"619761f1-b0a4-4b32-978d-5c3702133c7d","Type":"ContainerStarted","Data":"160937ee4fec9f1505646fa841d05488c201f823dec7fa41b39e4f84d4b21815"}
Sep 18 15:47:30.864177 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:30.864148    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v" event={"ID":"30914206-a17b-456a-89cc-decc238d701f","Type":"ContainerStarted","Data":"f2521d813fe32b1b1fe1643a77a5623b9fb9b70e0bda7b0144d56633c425a186"}
Sep 18 15:47:31.869717 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:31.869684    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-85fd65c94c-mx9mf" event={"ID":"619761f1-b0a4-4b32-978d-5c3702133c7d","Type":"ContainerStarted","Data":"8b4f20d9a8d5d7f16906945a921b3b91a314efc469af5e6ef6c1c4ce8feb7f1a"}
Sep 18 15:47:31.870150 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:31.869839    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-85fd65c94c-mx9mf"
Sep 18 15:47:31.871203 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:31.871179    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v" event={"ID":"30914206-a17b-456a-89cc-decc238d701f","Type":"ContainerStarted","Data":"1028e8d8912dc14cc7baaad4096bab0e6b4c12e7eb27d764439e1a5a7e9d2f40"}
Sep 18 15:47:31.871293 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:31.871225    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v"
Sep 18 15:47:31.889550 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:31.889506    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-85fd65c94c-mx9mf" podStartSLOduration=1.8705670319999999 podStartE2EDuration="2.889491873s" totalImagesPullingTime="1.018924841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:47:31.889292708 +0000 UTC m=+774.450524848" watchObservedRunningTime="2026-09-18 15:47:31.889491873 +0000 UTC m=+774.450724013"
Sep 18 15:47:31.905281 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:31.905240    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v" podStartSLOduration=1.8504800860000001 podStartE2EDuration="2.905225063s" totalImagesPullingTime="1.054744977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:47:31.904705727 +0000 UTC m=+774.465937893" watchObservedRunningTime="2026-09-18 15:47:31.905225063 +0000 UTC m=+774.466457202"
Sep 18 15:47:36.848721 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:36.848688    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92"
Sep 18 15:47:37.043427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:37.043396    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-65b57fbbf4-wj42c"]
Sep 18 15:47:37.046976 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:37.046956    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-65b57fbbf4-wj42c"
Sep 18 15:47:37.054366 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:37.054332    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-65b57fbbf4-wj42c"]
Sep 18 15:47:37.083294 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:37.083261    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4hb9\" (UniqueName: \"kubernetes.io/projected/a029d48e-b804-490a-8f0b-26da978a9a9a-kube-api-access-s4hb9\") pod \"ai-gateway-controller-65b57fbbf4-wj42c\" (UID: \"a029d48e-b804-490a-8f0b-26da978a9a9a\") " pod="opendatahub/ai-gateway-controller-65b57fbbf4-wj42c"
Sep 18 15:47:37.183678 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:37.183652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4hb9\" (UniqueName: \"kubernetes.io/projected/a029d48e-b804-490a-8f0b-26da978a9a9a-kube-api-access-s4hb9\") pod \"ai-gateway-controller-65b57fbbf4-wj42c\" (UID: \"a029d48e-b804-490a-8f0b-26da978a9a9a\") " pod="opendatahub/ai-gateway-controller-65b57fbbf4-wj42c"
Sep 18 15:47:37.192656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:37.192628    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4hb9\" (UniqueName: \"kubernetes.io/projected/a029d48e-b804-490a-8f0b-26da978a9a9a-kube-api-access-s4hb9\") pod \"ai-gateway-controller-65b57fbbf4-wj42c\" (UID: \"a029d48e-b804-490a-8f0b-26da978a9a9a\") " pod="opendatahub/ai-gateway-controller-65b57fbbf4-wj42c"
Sep 18 15:47:37.358046 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:37.358018    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-65b57fbbf4-wj42c"
Sep 18 15:47:37.624042 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:37.624016    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-65b57fbbf4-wj42c"]
Sep 18 15:47:37.626415 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:47:37.626386    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda029d48e_b804_490a_8f0b_26da978a9a9a.slice/crio-e092b0b51c2be69f4f939d41aa2cf1a9c67969a65a02ba8c57e8af9dc3710697 WatchSource:0}: Error finding container e092b0b51c2be69f4f939d41aa2cf1a9c67969a65a02ba8c57e8af9dc3710697: Status 404 returned error can't find the container with id e092b0b51c2be69f4f939d41aa2cf1a9c67969a65a02ba8c57e8af9dc3710697
Sep 18 15:47:37.897848 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:37.897775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-65b57fbbf4-wj42c" event={"ID":"a029d48e-b804-490a-8f0b-26da978a9a9a","Type":"ContainerStarted","Data":"e092b0b51c2be69f4f939d41aa2cf1a9c67969a65a02ba8c57e8af9dc3710697"}
Sep 18 15:47:38.903249 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:38.903222    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-65b57fbbf4-wj42c" event={"ID":"a029d48e-b804-490a-8f0b-26da978a9a9a","Type":"ContainerStarted","Data":"3d4237b3d364e82da80ecae5bb1026bfe8d5df5831876751ee338bb19d3005f2"}
Sep 18 15:47:38.903596 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:38.903355    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-65b57fbbf4-wj42c"
Sep 18 15:47:38.921392 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:38.921342    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-65b57fbbf4-wj42c" podStartSLOduration=1.5341153840000001 podStartE2EDuration="1.921326756s" totalImagesPullingTime="387.211372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:47:38.920471231 +0000 UTC m=+781.481703396" watchObservedRunningTime="2026-09-18 15:47:38.921326756 +0000 UTC m=+781.482558894"
Sep 18 15:47:42.877568 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:42.877539    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-85fd65c94c-mx9mf"
Sep 18 15:47:42.877979 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:42.877617    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-76f657cdc4-z8d9v"
Sep 18 15:47:49.910204 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:49.910175    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-65b57fbbf4-wj42c"
Sep 18 15:47:49.958579 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:49.958551    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-586bd8bd6-8rr92"]
Sep 18 15:47:49.958773 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:49.958743    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92" podUID="0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2" containerName="manager" containerID="cri-o://8677ab41e0130070e26543aff05fbfcf319c628744c0c1cbd263daec605f0ecf" gracePeriod=30
Sep 18 15:47:50.197194 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:50.197170    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92"
Sep 18 15:47:50.291345 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:50.291318    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfsr4\" (UniqueName: \"kubernetes.io/projected/0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2-kube-api-access-gfsr4\") pod \"0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2\" (UID: \"0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2\") "
Sep 18 15:47:50.293395 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:50.293368    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2-kube-api-access-gfsr4" (OuterVolumeSpecName: "kube-api-access-gfsr4") pod "0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2" (UID: "0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2"). InnerVolumeSpecName "kube-api-access-gfsr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:47:50.392078 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:50.392054    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfsr4\" (UniqueName: \"kubernetes.io/projected/0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2-kube-api-access-gfsr4\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:47:50.959518 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:50.959480    2559 generic.go:356] "Generic (PLEG): container finished" podID="0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2" containerID="8677ab41e0130070e26543aff05fbfcf319c628744c0c1cbd263daec605f0ecf" exitCode=0
Sep 18 15:47:50.959518 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:50.959523    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92" event={"ID":"0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2","Type":"ContainerDied","Data":"8677ab41e0130070e26543aff05fbfcf319c628744c0c1cbd263daec605f0ecf"}
Sep 18 15:47:50.959992 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:50.959530    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92"
Sep 18 15:47:50.959992 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:50.959582    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-586bd8bd6-8rr92" event={"ID":"0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2","Type":"ContainerDied","Data":"01e565e73185a80de99d7942847031100fa9d445cb8e9c12ad3d8bd5d018cbbc"}
Sep 18 15:47:50.959992 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:50.959602    2559 scope.go:117] "RemoveContainer" containerID="8677ab41e0130070e26543aff05fbfcf319c628744c0c1cbd263daec605f0ecf"
Sep 18 15:47:50.967534 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:50.967517    2559 scope.go:117] "RemoveContainer" containerID="8677ab41e0130070e26543aff05fbfcf319c628744c0c1cbd263daec605f0ecf"
Sep 18 15:47:50.967773 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:47:50.967750    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8677ab41e0130070e26543aff05fbfcf319c628744c0c1cbd263daec605f0ecf\": container with ID starting with 8677ab41e0130070e26543aff05fbfcf319c628744c0c1cbd263daec605f0ecf not found: ID does not exist" containerID="8677ab41e0130070e26543aff05fbfcf319c628744c0c1cbd263daec605f0ecf"
Sep 18 15:47:50.967817 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:50.967785    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8677ab41e0130070e26543aff05fbfcf319c628744c0c1cbd263daec605f0ecf"} err="failed to get container status \"8677ab41e0130070e26543aff05fbfcf319c628744c0c1cbd263daec605f0ecf\": rpc error: code = NotFound desc = could not find container \"8677ab41e0130070e26543aff05fbfcf319c628744c0c1cbd263daec605f0ecf\": container with ID starting with 8677ab41e0130070e26543aff05fbfcf319c628744c0c1cbd263daec605f0ecf not found: ID does not exist"
Sep 18 15:47:50.980498 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:50.980476    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-586bd8bd6-8rr92"]
Sep 18 15:47:50.984104 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:50.984084    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-586bd8bd6-8rr92"]
Sep 18 15:47:51.119881 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:51.119838    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:47:51.448432 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:51.448401    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:47:51.743204 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:51.743130    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2" path="/var/lib/kubelet/pods/0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2/volumes"
Sep 18 15:47:52.561758 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:52.561729    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:47:52.659714 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:52.659681    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:47:53.610440 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:53.610399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:47:53.705012 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:47:53.704982    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:08.234427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.234389    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:08.291399 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.291364    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:08.575295 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.575190    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:08.622584 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.622551    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"]
Sep 18 15:48:08.622973 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.622959    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2" containerName="manager"
Sep 18 15:48:08.623030 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.622975    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2" containerName="manager"
Sep 18 15:48:08.623069 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.623036    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bf9ebc3-a24f-424b-8b17-6f867ff0c0e2" containerName="manager"
Sep 18 15:48:08.628300 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.628275    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.631109 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.631079    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 15:48:08.631613 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.631596    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9792ff-openshift-default-dockercfg-5klgd\""
Sep 18 15:48:08.643680 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.643650    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"]
Sep 18 15:48:08.669706 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.669661    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:08.739974 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.739944    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-istio-data\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.740134 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.739985    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-workload-socket\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.740134 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.740005    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df52776c-9cbe-417e-ab2a-4164812cb020-istio-podinfo\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.740134 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.740102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-istio-envoy\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.740297 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.740156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-credential-socket\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.740297 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.740189    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df52776c-9cbe-417e-ab2a-4164812cb020-istiod-ca-cert\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.740297 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.740225    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9g2q\" (UniqueName: \"kubernetes.io/projected/df52776c-9cbe-417e-ab2a-4164812cb020-kube-api-access-c9g2q\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.740297 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.740280    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-workload-certs\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.740450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.740304    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df52776c-9cbe-417e-ab2a-4164812cb020-istio-token\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.769264 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.769237    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:08.841351 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.841263    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-workload-certs\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.841351 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.841297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df52776c-9cbe-417e-ab2a-4164812cb020-istio-token\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.841572 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.841416    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-istio-data\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.841572 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.841452    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-workload-socket\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.841572 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.841480    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df52776c-9cbe-417e-ab2a-4164812cb020-istio-podinfo\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.841728 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.841571    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-istio-envoy\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.841728 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.841614    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-credential-socket\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.841728 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.841652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df52776c-9cbe-417e-ab2a-4164812cb020-istiod-ca-cert\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.841728 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.841681    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-workload-certs\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.841728 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.841723    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9g2q\" (UniqueName: \"kubernetes.io/projected/df52776c-9cbe-417e-ab2a-4164812cb020-kube-api-access-c9g2q\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.842026 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.841797    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-istio-data\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.842026 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.841906    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-workload-socket\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.842288 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.842267    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-credential-socket\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.842394 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.842377    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df52776c-9cbe-417e-ab2a-4164812cb020-istiod-ca-cert\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.844346 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.844324    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-istio-envoy\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.844702 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.844685    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df52776c-9cbe-417e-ab2a-4164812cb020-istio-podinfo\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.850071 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.850046    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df52776c-9cbe-417e-ab2a-4164812cb020-istio-token\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.850694 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.850665    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9g2q\" (UniqueName: \"kubernetes.io/projected/df52776c-9cbe-417e-ab2a-4164812cb020-kube-api-access-c9g2q\") pod \"e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") " pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:08.869160 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.869133    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:08.938719 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:08.938688    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:09.212057 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:09.212023    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"]
Sep 18 15:48:09.214516 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:48:09.214449    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf52776c_9cbe_417e_ab2a_4164812cb020.slice/crio-a6a47c2a2aac4d87f631c487fdda486a417dcf29cbeb763f081307e025cd41b8 WatchSource:0}: Error finding container a6a47c2a2aac4d87f631c487fdda486a417dcf29cbeb763f081307e025cd41b8: Status 404 returned error can't find the container with id a6a47c2a2aac4d87f631c487fdda486a417dcf29cbeb763f081307e025cd41b8
Sep 18 15:48:09.269164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:09.269134    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:09.364395 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:09.364354    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:09.625368 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:09.625290    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:09.725115 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:09.725079    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:10.044130 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:10.044091    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc" event={"ID":"df52776c-9cbe-417e-ab2a-4164812cb020","Type":"ContainerStarted","Data":"a6a47c2a2aac4d87f631c487fdda486a417dcf29cbeb763f081307e025cd41b8"}
Sep 18 15:48:10.056948 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:10.056922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:11.574897 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:11.574840    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:11.692223 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:11.692187    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:11.776390 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:11.776351    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:12.032829 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:12.032779    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:48:12.032959 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:12.032884    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:48:12.032959 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:12.032915    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:48:12.747444 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:12.747400    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:12.841250 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:12.841215    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:13.060243 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.060150    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc" event={"ID":"df52776c-9cbe-417e-ab2a-4164812cb020","Type":"ContainerStarted","Data":"0ec2dc228e6a5a223ca1ac2445c9c898c0535db1be7e6b32efe84fd97a7974a4"}
Sep 18 15:48:13.088164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.088105    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc" podStartSLOduration=2.27226896 podStartE2EDuration="5.088090827s" totalImagesPullingTime="2.815821867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:13.086905075 +0000 UTC m=+815.648137220" watchObservedRunningTime="2026-09-18 15:48:13.088090827 +0000 UTC m=+815.649322965"
Sep 18 15:48:13.261305 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.261275    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:13.363791 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.363704    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:13.381056 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.381026    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"]
Sep 18 15:48:13.383813 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.383791    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"
Sep 18 15:48:13.386672 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.386654    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a1f773\""
Sep 18 15:48:13.386778 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.386655    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a1f773-dockercfg-299kc\""
Sep 18 15:48:13.397258 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.397233    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"]
Sep 18 15:48:13.450615 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.450577    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"]
Sep 18 15:48:13.453034 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.453020    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"
Sep 18 15:48:13.463543 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.463514    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"]
Sep 18 15:48:13.490669 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.490636    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2cdb485-d211-4db7-9c80-0b8a3db81039-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz\" (UID: \"f2cdb485-d211-4db7-9c80-0b8a3db81039\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"
Sep 18 15:48:13.490836 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.490699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rntj\" (UniqueName: \"kubernetes.io/projected/f2cdb485-d211-4db7-9c80-0b8a3db81039-kube-api-access-9rntj\") pod \"payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz\" (UID: \"f2cdb485-d211-4db7-9c80-0b8a3db81039\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"
Sep 18 15:48:13.591269 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.591230    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2cdb485-d211-4db7-9c80-0b8a3db81039-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz\" (UID: \"f2cdb485-d211-4db7-9c80-0b8a3db81039\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"
Sep 18 15:48:13.591443 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.591295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d83550d-db6b-4d86-9df9-bcea236c32b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb\" (UID: \"8d83550d-db6b-4d86-9df9-bcea236c32b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"
Sep 18 15:48:13.591443 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.591316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rntj\" (UniqueName: \"kubernetes.io/projected/f2cdb485-d211-4db7-9c80-0b8a3db81039-kube-api-access-9rntj\") pod \"payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz\" (UID: \"f2cdb485-d211-4db7-9c80-0b8a3db81039\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"
Sep 18 15:48:13.591443 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.591351    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpgrv\" (UniqueName: \"kubernetes.io/projected/8d83550d-db6b-4d86-9df9-bcea236c32b1-kube-api-access-qpgrv\") pod \"payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb\" (UID: \"8d83550d-db6b-4d86-9df9-bcea236c32b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"
Sep 18 15:48:13.591964 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.591947    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2cdb485-d211-4db7-9c80-0b8a3db81039-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz\" (UID: \"f2cdb485-d211-4db7-9c80-0b8a3db81039\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"
Sep 18 15:48:13.600588 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.600564    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rntj\" (UniqueName: \"kubernetes.io/projected/f2cdb485-d211-4db7-9c80-0b8a3db81039-kube-api-access-9rntj\") pod \"payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz\" (UID: \"f2cdb485-d211-4db7-9c80-0b8a3db81039\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"
Sep 18 15:48:13.692406 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.692373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpgrv\" (UniqueName: \"kubernetes.io/projected/8d83550d-db6b-4d86-9df9-bcea236c32b1-kube-api-access-qpgrv\") pod \"payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb\" (UID: \"8d83550d-db6b-4d86-9df9-bcea236c32b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"
Sep 18 15:48:13.692553 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.692463    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d83550d-db6b-4d86-9df9-bcea236c32b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb\" (UID: \"8d83550d-db6b-4d86-9df9-bcea236c32b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"
Sep 18 15:48:13.693126 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.693105    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d83550d-db6b-4d86-9df9-bcea236c32b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb\" (UID: \"8d83550d-db6b-4d86-9df9-bcea236c32b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"
Sep 18 15:48:13.694289 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.694273    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"
Sep 18 15:48:13.702507 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.702489    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpgrv\" (UniqueName: \"kubernetes.io/projected/8d83550d-db6b-4d86-9df9-bcea236c32b1-kube-api-access-qpgrv\") pod \"payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb\" (UID: \"8d83550d-db6b-4d86-9df9-bcea236c32b1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"
Sep 18 15:48:13.764232 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.764199    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"
Sep 18 15:48:13.803078 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.803040    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:13.903179 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.903143    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:13.939597 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.939562    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:13.944577 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.944514    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:13.965525 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:13.965503    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"]
Sep 18 15:48:13.968132 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:48:13.968110    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cdb485_d211_4db7_9c80_0b8a3db81039.slice/crio-3a0f9ead60c899d2b5c31b210744a22dd2eec792a1db2d9ab17690e4f5373fd2 WatchSource:0}: Error finding container 3a0f9ead60c899d2b5c31b210744a22dd2eec792a1db2d9ab17690e4f5373fd2: Status 404 returned error can't find the container with id 3a0f9ead60c899d2b5c31b210744a22dd2eec792a1db2d9ab17690e4f5373fd2
Sep 18 15:48:14.042558 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.042499    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"]
Sep 18 15:48:14.045078 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:48:14.045053    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d83550d_db6b_4d86_9df9_bcea236c32b1.slice/crio-cab7a7e08cdd2fbfb198547f7d1a9bcfa4447df12f13a4c57dc388338d0796b9 WatchSource:0}: Error finding container cab7a7e08cdd2fbfb198547f7d1a9bcfa4447df12f13a4c57dc388338d0796b9: Status 404 returned error can't find the container with id cab7a7e08cdd2fbfb198547f7d1a9bcfa4447df12f13a4c57dc388338d0796b9
Sep 18 15:48:14.067700 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.067675    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb" event={"ID":"8d83550d-db6b-4d86-9df9-bcea236c32b1","Type":"ContainerStarted","Data":"cab7a7e08cdd2fbfb198547f7d1a9bcfa4447df12f13a4c57dc388338d0796b9"}
Sep 18 15:48:14.069100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.069074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz" event={"ID":"f2cdb485-d211-4db7-9c80-0b8a3db81039","Type":"ContainerStarted","Data":"4337b80315ef21ce78f4b18180982dd249913ff86a09c51a5fdf4b118f6d11d4"}
Sep 18 15:48:14.069201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.069123    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz" event={"ID":"f2cdb485-d211-4db7-9c80-0b8a3db81039","Type":"ContainerStarted","Data":"3a0f9ead60c899d2b5c31b210744a22dd2eec792a1db2d9ab17690e4f5373fd2"}
Sep 18 15:48:14.069307 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.069289    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:14.070333 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.070316    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:14.089727 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.089684    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz" podStartSLOduration=1.089669921 podStartE2EDuration="1.089669921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:14.088268004 +0000 UTC m=+816.649500165" watchObservedRunningTime="2026-09-18 15:48:14.089669921 +0000 UTC m=+816.650902059"
Sep 18 15:48:14.176665 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.176634    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:14.411333 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.411237    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:14.502573 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.502539    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:14.607128 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.607094    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"]
Sep 18 15:48:14.610350 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.610327    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"
Sep 18 15:48:14.613378 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.613355    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-998130\""
Sep 18 15:48:14.613502 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.613356    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-998130-dockercfg-qbl4z\""
Sep 18 15:48:14.624542 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.624513    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"]
Sep 18 15:48:14.802240 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.802199    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6573b858-2906-4a09-a4e3-55ea7238dbb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk\" (UID: \"6573b858-2906-4a09-a4e3-55ea7238dbb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"
Sep 18 15:48:14.802661 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.802280    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfwk\" (UniqueName: \"kubernetes.io/projected/6573b858-2906-4a09-a4e3-55ea7238dbb3-kube-api-access-pkfwk\") pod \"payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk\" (UID: \"6573b858-2906-4a09-a4e3-55ea7238dbb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"
Sep 18 15:48:14.903186 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.903136    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6573b858-2906-4a09-a4e3-55ea7238dbb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk\" (UID: \"6573b858-2906-4a09-a4e3-55ea7238dbb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"
Sep 18 15:48:14.903373 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.903218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfwk\" (UniqueName: \"kubernetes.io/projected/6573b858-2906-4a09-a4e3-55ea7238dbb3-kube-api-access-pkfwk\") pod \"payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk\" (UID: \"6573b858-2906-4a09-a4e3-55ea7238dbb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"
Sep 18 15:48:14.903965 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.903947    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6573b858-2906-4a09-a4e3-55ea7238dbb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk\" (UID: \"6573b858-2906-4a09-a4e3-55ea7238dbb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"
Sep 18 15:48:14.913546 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.913520    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfwk\" (UniqueName: \"kubernetes.io/projected/6573b858-2906-4a09-a4e3-55ea7238dbb3-kube-api-access-pkfwk\") pod \"payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk\" (UID: \"6573b858-2906-4a09-a4e3-55ea7238dbb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"
Sep 18 15:48:14.921321 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.921299    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"
Sep 18 15:48:14.999984 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:14.999950    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:15.076421 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:15.076320    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb" event={"ID":"8d83550d-db6b-4d86-9df9-bcea236c32b1","Type":"ContainerStarted","Data":"94cd1a33b0740dc4e7c0267789649bb4bc70c139477860f4aba5a82315b01d3f"}
Sep 18 15:48:15.076807 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:15.076789    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"
Sep 18 15:48:15.100775 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:15.100706    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb" podStartSLOduration=2.100682906 podStartE2EDuration="2.100682906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:15.099869778 +0000 UTC m=+817.661101911" watchObservedRunningTime="2026-09-18 15:48:15.100682906 +0000 UTC m=+817.661915048"
Sep 18 15:48:15.206955 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:15.206931    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"]
Sep 18 15:48:15.208630 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:48:15.208602    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6573b858_2906_4a09_a4e3_55ea7238dbb3.slice/crio-86b35f16277720db7905b8d72f2ed1ee45bddad6822fbc1431799932d832701d WatchSource:0}: Error finding container 86b35f16277720db7905b8d72f2ed1ee45bddad6822fbc1431799932d832701d: Status 404 returned error can't find the container with id 86b35f16277720db7905b8d72f2ed1ee45bddad6822fbc1431799932d832701d
Sep 18 15:48:15.366713 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:15.366596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:15.477493 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:15.477438    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:15.924183 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:15.924145    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:16.083612 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:16.083579    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk" event={"ID":"6573b858-2906-4a09-a4e3-55ea7238dbb3","Type":"ContainerStarted","Data":"da4dfe76870bdd3a41ab6f2c61fedf6247337b7edbffa0eb005a5b3d30757912"}
Sep 18 15:48:16.083820 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:16.083635    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk" event={"ID":"6573b858-2906-4a09-a4e3-55ea7238dbb3","Type":"ContainerStarted","Data":"86b35f16277720db7905b8d72f2ed1ee45bddad6822fbc1431799932d832701d"}
Sep 18 15:48:16.084148 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:16.084128    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"
Sep 18 15:48:16.117616 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:16.117562    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk" podStartSLOduration=2.11754635 podStartE2EDuration="2.11754635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:16.115813161 +0000 UTC m=+818.677045297" watchObservedRunningTime="2026-09-18 15:48:16.11754635 +0000 UTC m=+818.678778489"
Sep 18 15:48:16.713567 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:16.713530    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:17.088112 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:17.088081    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"
Sep 18 15:48:17.703960 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:17.703902    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:17.820183 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:17.820152    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:17.835897 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:17.835870    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:48:18.077222 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:18.077131    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:18.498127 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:18.498098    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:18.596245 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:18.596209    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:18.904361 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:18.904270    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:19.031900 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.031866    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:19.144637 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.144601    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:19.434055 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.433992    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"]
Sep 18 15:48:19.437030 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.437015    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.440542 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.440520    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a8802cb2-openshift-default-dockercfg-lcn7z\""
Sep 18 15:48:19.470337 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.470308    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"]
Sep 18 15:48:19.496645 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.496615    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:19.542669 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.542631    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istiod-ca-cert\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.542669 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.542670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-workload-certs\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.543160 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.542696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg8wt\" (UniqueName: \"kubernetes.io/projected/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-kube-api-access-rg8wt\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.543160 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.542771    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-data\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.543160 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.542793    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-token\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.543160 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.542816    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-envoy\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.543160 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.542941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-credential-socket\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.543160 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.542969    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-podinfo\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.543160 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.542993    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-workload-socket\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.644349 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.644313    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-data\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.644349 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.644352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-token\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.644577 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.644374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-envoy\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.644577 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.644407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-credential-socket\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.644577 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.644445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-podinfo\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.644577 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.644475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-workload-socket\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.644577 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.644534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istiod-ca-cert\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.644577 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.644564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-workload-certs\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.644960 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.644588    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg8wt\" (UniqueName: \"kubernetes.io/projected/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-kube-api-access-rg8wt\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.644960 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.644774    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-data\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.644960 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.644916    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-credential-socket\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.645105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.645036    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-workload-socket\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.645168 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.645136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-workload-certs\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.645331 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.645309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istiod-ca-cert\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.646788 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.646766    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-envoy\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.647100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.647082    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-podinfo\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.653403 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.653378    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-token\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.654697 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.654671    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg8wt\" (UniqueName: \"kubernetes.io/projected/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-kube-api-access-rg8wt\") pod \"e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") " pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:19.748893 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:19.748789    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:20.027591 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:20.027564    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"]
Sep 18 15:48:20.029677 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:48:20.029646    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebe17ad_2f8b_41cc_a090_6af5c5bc6e81.slice/crio-3d599e40423e477e91210506a338ef64015d8cf850cad53e4f7177c295b5776a WatchSource:0}: Error finding container 3d599e40423e477e91210506a338ef64015d8cf850cad53e4f7177c295b5776a: Status 404 returned error can't find the container with id 3d599e40423e477e91210506a338ef64015d8cf850cad53e4f7177c295b5776a
Sep 18 15:48:20.032814 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:20.032782    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:48:20.032933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:20.032849    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:48:20.032933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:20.032904    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:48:20.035791 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:20.035768    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:20.102570 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:20.102545    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z" event={"ID":"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81","Type":"ContainerStarted","Data":"95971c850cae37c07d9e451a67955076f7027cb47694de8f3328f5a4b8a68105"}
Sep 18 15:48:20.102690 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:20.102600    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z" event={"ID":"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81","Type":"ContainerStarted","Data":"3d599e40423e477e91210506a338ef64015d8cf850cad53e4f7177c295b5776a"}
Sep 18 15:48:20.129414 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:20.129350    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z" podStartSLOduration=1.129331519 podStartE2EDuration="1.129331519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:20.12728971 +0000 UTC m=+822.688521865" watchObservedRunningTime="2026-09-18 15:48:20.129331519 +0000 UTC m=+822.690563657"
Sep 18 15:48:20.749489 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:20.748932    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:20.892655 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:20.892622    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:21.532205 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:21.532163    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:21.633139 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:21.633095    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:21.755314 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:21.755287    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:22.060474 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:22.060437    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:22.112108 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:22.112077    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:22.112918 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:22.112900    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:23.035873 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:23.035807    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:23.876669 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:23.876635    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:24.715671 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:24.715622    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:24.815206 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:24.815171    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:25.046594 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.046508    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:25.146059 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.146012    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:25.281138 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.281103    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:25.442080 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.442043    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:25.568246 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.568210    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:25.665393 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.665357    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"]
Sep 18 15:48:25.669036 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.669016    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"
Sep 18 15:48:25.672682 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.672663    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a8802cb2-dockercfg-kp2cg\""
Sep 18 15:48:25.672785 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.672722    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a8802cb2\""
Sep 18 15:48:25.681580 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.681554    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"]
Sep 18 15:48:25.700920 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.700826    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4px7\" (UniqueName: \"kubernetes.io/projected/54c8096d-2fc9-420b-a6a5-9777962919aa-kube-api-access-d4px7\") pod \"payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5\" (UID: \"54c8096d-2fc9-420b-a6a5-9777962919aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"
Sep 18 15:48:25.700920 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.700904    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c8096d-2fc9-420b-a6a5-9777962919aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5\" (UID: \"54c8096d-2fc9-420b-a6a5-9777962919aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"
Sep 18 15:48:25.712134 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.712102    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"]
Sep 18 15:48:25.715484 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.715468    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"
Sep 18 15:48:25.749988 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.749952    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"]
Sep 18 15:48:25.802226 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.802194    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82ab850-b1a6-4298-ad8a-4d5a0d1c7513-plugins-config-volume\") pod \"payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68\" (UID: \"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513\") " pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"
Sep 18 15:48:25.802387 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.802233    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmk4n\" (UniqueName: \"kubernetes.io/projected/a82ab850-b1a6-4298-ad8a-4d5a0d1c7513-kube-api-access-zmk4n\") pod \"payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68\" (UID: \"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513\") " pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"
Sep 18 15:48:25.802387 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.802310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4px7\" (UniqueName: \"kubernetes.io/projected/54c8096d-2fc9-420b-a6a5-9777962919aa-kube-api-access-d4px7\") pod \"payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5\" (UID: \"54c8096d-2fc9-420b-a6a5-9777962919aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"
Sep 18 15:48:25.802387 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.802346    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c8096d-2fc9-420b-a6a5-9777962919aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5\" (UID: \"54c8096d-2fc9-420b-a6a5-9777962919aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"
Sep 18 15:48:25.803068 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.803050    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c8096d-2fc9-420b-a6a5-9777962919aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5\" (UID: \"54c8096d-2fc9-420b-a6a5-9777962919aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"
Sep 18 15:48:25.817646 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.817616    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4px7\" (UniqueName: \"kubernetes.io/projected/54c8096d-2fc9-420b-a6a5-9777962919aa-kube-api-access-d4px7\") pod \"payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5\" (UID: \"54c8096d-2fc9-420b-a6a5-9777962919aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"
Sep 18 15:48:25.903630 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.903595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82ab850-b1a6-4298-ad8a-4d5a0d1c7513-plugins-config-volume\") pod \"payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68\" (UID: \"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513\") " pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"
Sep 18 15:48:25.903630 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.903634    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmk4n\" (UniqueName: \"kubernetes.io/projected/a82ab850-b1a6-4298-ad8a-4d5a0d1c7513-kube-api-access-zmk4n\") pod \"payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68\" (UID: \"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513\") " pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"
Sep 18 15:48:25.904311 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.904288    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82ab850-b1a6-4298-ad8a-4d5a0d1c7513-plugins-config-volume\") pod \"payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68\" (UID: \"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513\") " pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"
Sep 18 15:48:25.914587 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.914562    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmk4n\" (UniqueName: \"kubernetes.io/projected/a82ab850-b1a6-4298-ad8a-4d5a0d1c7513-kube-api-access-zmk4n\") pod \"payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68\" (UID: \"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513\") " pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"
Sep 18 15:48:25.980488 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:25.980413    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"
Sep 18 15:48:26.025750 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:26.025719    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"
Sep 18 15:48:26.086530 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:26.086108    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"
Sep 18 15:48:26.094782 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:26.094670    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:26.230874 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:26.230771    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:26.251483 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:26.251455    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"]
Sep 18 15:48:26.253247 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:48:26.253221    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c8096d_2fc9_420b_a6a5_9777962919aa.slice/crio-8bfec7d419b377c85438213a8aa7b1aedfde652670a5b39f4fc721da015ee2de WatchSource:0}: Error finding container 8bfec7d419b377c85438213a8aa7b1aedfde652670a5b39f4fc721da015ee2de: Status 404 returned error can't find the container with id 8bfec7d419b377c85438213a8aa7b1aedfde652670a5b39f4fc721da015ee2de
Sep 18 15:48:26.313648 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:26.313624    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"]
Sep 18 15:48:26.314496 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:48:26.314461    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82ab850_b1a6_4298_ad8a_4d5a0d1c7513.slice/crio-d951ee1b151c8976138c9689e73f27221b7ed64b50114333a8958ca964219cbc WatchSource:0}: Error finding container d951ee1b151c8976138c9689e73f27221b7ed64b50114333a8958ca964219cbc: Status 404 returned error can't find the container with id d951ee1b151c8976138c9689e73f27221b7ed64b50114333a8958ca964219cbc
Sep 18 15:48:26.530146 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:26.530052    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:26.843922 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:26.843813    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:26.968344 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:26.968307    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:27.072766 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:27.072739    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:27.091067 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:27.091040    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"
Sep 18 15:48:27.141561 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:27.141456    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68" event={"ID":"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513","Type":"ContainerStarted","Data":"0458ffdcac14a15d9fa14aff8d3496530f12c184b039c5b72903e75f16803ec9"}
Sep 18 15:48:27.141561 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:27.141525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68" event={"ID":"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513","Type":"ContainerStarted","Data":"d951ee1b151c8976138c9689e73f27221b7ed64b50114333a8958ca964219cbc"}
Sep 18 15:48:27.141561 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:27.141553    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"
Sep 18 15:48:27.146697 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:27.146665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5" event={"ID":"54c8096d-2fc9-420b-a6a5-9777962919aa","Type":"ContainerStarted","Data":"c14bc35d070194034f301d7fb9ed205c1c849dc2c08d831578928a642a1a34dd"}
Sep 18 15:48:27.146896 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:27.146715    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5" event={"ID":"54c8096d-2fc9-420b-a6a5-9777962919aa","Type":"ContainerStarted","Data":"8bfec7d419b377c85438213a8aa7b1aedfde652670a5b39f4fc721da015ee2de"}
Sep 18 15:48:27.146896 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:27.146738    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"
Sep 18 15:48:27.160507 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:27.160458    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68" podStartSLOduration=2.16044194 podStartE2EDuration="2.16044194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:27.159603527 +0000 UTC m=+829.720835668" watchObservedRunningTime="2026-09-18 15:48:27.16044194 +0000 UTC m=+829.721674069"
Sep 18 15:48:27.177478 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:27.177420    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5" podStartSLOduration=2.177395512 podStartE2EDuration="2.177395512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:27.176414723 +0000 UTC m=+829.737646923" watchObservedRunningTime="2026-09-18 15:48:27.177395512 +0000 UTC m=+829.738627654"
Sep 18 15:48:27.685986 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:27.685943    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:28.095118 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:28.095028    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"
Sep 18 15:48:28.654747 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:28.654700    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:29.641838 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:29.641780    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:30.054501 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:30.054467    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:30.419082 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:30.419003    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:30.559721 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:30.559628    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:30.709933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:30.709888    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:31.396627 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:31.396583    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:31.597516 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:31.597482    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:31.841505 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:31.841468    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:31.939530 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:31.939492    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:32.493240 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:32.493197    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:32.645376 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:32.645342    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:32.793353 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:32.793269    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:33.955412 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:33.955377    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:34.129887 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:34.129828    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:34.263981 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:34.263889    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:34.390446 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:34.390413    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:34.543171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:34.543075    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:35.005394 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:35.005366    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:35.987222 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:35.987171    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:36.091026 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.090989    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:36.545766 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.545730    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:36.647476 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.647435    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"]
Sep 18 15:48:36.649622 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.649590    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:36.652177 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.652157    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"
Sep 18 15:48:36.654902 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.654879    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-70c6df\""
Sep 18 15:48:36.669080 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.669055    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"]
Sep 18 15:48:36.710097 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.710067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94b76d4-ea0f-486d-b9ca-f1375e72bcf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2\" (UID: \"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"
Sep 18 15:48:36.710260 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.710191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj79z\" (UniqueName: \"kubernetes.io/projected/b94b76d4-ea0f-486d-b9ca-f1375e72bcf4-kube-api-access-lj79z\") pod \"payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2\" (UID: \"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"
Sep 18 15:48:36.711619 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.711598    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9"]
Sep 18 15:48:36.715308 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.715276    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9"
Sep 18 15:48:36.718117 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.718093    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 15:48:36.718422 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.718399    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 15:48:36.724201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.724172    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9"]
Sep 18 15:48:36.747962 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.747921    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:36.811033 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.810931    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj79z\" (UniqueName: \"kubernetes.io/projected/b94b76d4-ea0f-486d-b9ca-f1375e72bcf4-kube-api-access-lj79z\") pod \"payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2\" (UID: \"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"
Sep 18 15:48:36.811033 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.810993    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1492ab45-034b-48c7-a97c-950cfc340a67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9\" (UID: \"1492ab45-034b-48c7-a97c-950cfc340a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9"
Sep 18 15:48:36.811251 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.811057    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94b76d4-ea0f-486d-b9ca-f1375e72bcf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2\" (UID: \"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"
Sep 18 15:48:36.811797 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.811778    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94b76d4-ea0f-486d-b9ca-f1375e72bcf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2\" (UID: \"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"
Sep 18 15:48:36.820217 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.820191    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj79z\" (UniqueName: \"kubernetes.io/projected/b94b76d4-ea0f-486d-b9ca-f1375e72bcf4-kube-api-access-lj79z\") pod \"payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2\" (UID: \"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"
Sep 18 15:48:36.911783 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.911746    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1492ab45-034b-48c7-a97c-950cfc340a67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9\" (UID: \"1492ab45-034b-48c7-a97c-950cfc340a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9"
Sep 18 15:48:36.918571 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.912974    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1492ab45-034b-48c7-a97c-950cfc340a67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9\" (UID: \"1492ab45-034b-48c7-a97c-950cfc340a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9"
Sep 18 15:48:36.964608 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:36.964565    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"
Sep 18 15:48:37.028496 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:37.028152    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9"
Sep 18 15:48:37.201990 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:37.201946    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:37.250048 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:37.250002    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"]
Sep 18 15:48:37.251848 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:48:37.251817    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94b76d4_ea0f_486d_b9ca_f1375e72bcf4.slice/crio-308bef90befd308e19bc46f46573fd3403d777d4fd5bed45ac56639c3d149356 WatchSource:0}: Error finding container 308bef90befd308e19bc46f46573fd3403d777d4fd5bed45ac56639c3d149356: Status 404 returned error can't find the container with id 308bef90befd308e19bc46f46573fd3403d777d4fd5bed45ac56639c3d149356
Sep 18 15:48:37.312630 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:37.312595    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9"]
Sep 18 15:48:37.314410 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:48:37.314374    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1492ab45_034b_48c7_a97c_950cfc340a67.slice/crio-bccffef4768e4a58270c53eb7324d1b4252c328f38d669d1394672171ef1db9c WatchSource:0}: Error finding container bccffef4768e4a58270c53eb7324d1b4252c328f38d669d1394672171ef1db9c: Status 404 returned error can't find the container with id bccffef4768e4a58270c53eb7324d1b4252c328f38d669d1394672171ef1db9c
Sep 18 15:48:37.412825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:37.412729    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:48:38.154482 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.154406    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"
Sep 18 15:48:38.154482 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.154483    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"
Sep 18 15:48:38.214604 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.214572    2559 generic.go:356] "Generic (PLEG): container finished" podID="df52776c-9cbe-417e-ab2a-4164812cb020" containerID="0ec2dc228e6a5a223ca1ac2445c9c898c0535db1be7e6b32efe84fd97a7974a4" exitCode=0
Sep 18 15:48:38.215058 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.214607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc" event={"ID":"df52776c-9cbe-417e-ab2a-4164812cb020","Type":"ContainerDied","Data":"0ec2dc228e6a5a223ca1ac2445c9c898c0535db1be7e6b32efe84fd97a7974a4"}
Sep 18 15:48:38.215131 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.215106    2559 scope.go:117] "RemoveContainer" containerID="0ec2dc228e6a5a223ca1ac2445c9c898c0535db1be7e6b32efe84fd97a7974a4"
Sep 18 15:48:38.216123 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.216089    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9" event={"ID":"1492ab45-034b-48c7-a97c-950cfc340a67","Type":"ContainerStarted","Data":"bccffef4768e4a58270c53eb7324d1b4252c328f38d669d1394672171ef1db9c"}
Sep 18 15:48:38.217880 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.217817    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:48:38.217994 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.217925    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2" event={"ID":"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4","Type":"ContainerStarted","Data":"9b4406676dc5ecffa2873b3e28636928c84470f1e94c863f38091c7edacda5c9"}
Sep 18 15:48:38.217994 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.217941    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:48:38.217994 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.217966    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2" event={"ID":"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4","Type":"ContainerStarted","Data":"308bef90befd308e19bc46f46573fd3403d777d4fd5bed45ac56639c3d149356"}
Sep 18 15:48:38.218182 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.217999    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:48:38.218275 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.218257    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"
Sep 18 15:48:38.256674 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.256628    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2" podStartSLOduration=2.256610589 podStartE2EDuration="2.256610589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:38.256205906 +0000 UTC m=+840.817438072" watchObservedRunningTime="2026-09-18 15:48:38.256610589 +0000 UTC m=+840.817842796"
Sep 18 15:48:38.276252 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.276223    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:38.990256 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:38.990186    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:39.227208 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:39.227171    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc" event={"ID":"df52776c-9cbe-417e-ab2a-4164812cb020","Type":"ContainerStarted","Data":"eb97210ade1a97851dfca4f15912a41cbdd3477659439f5433d02669dfd56e61"}
Sep 18 15:48:39.782880 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:39.778357    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:39.939413 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:39.939361    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:39.940584 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:39.940553    2559 patch_prober.go:28] interesting pod/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 18 15:48:39.940729 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:39.940610    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc" podUID="df52776c-9cbe-417e-ab2a-4164812cb020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 18 15:48:40.241193 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.241075    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"]
Sep 18 15:48:40.249197 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.249174    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.258273 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.256179    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9792ff\"/\"simulator-w3-9792ff-kserve-self-signed-certs\""
Sep 18 15:48:40.258273 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.257521    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9792ff\"/\"kube-root-ca.crt\""
Sep 18 15:48:40.258519 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.258393    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9792ff\"/\"openshift-service-ca.crt\""
Sep 18 15:48:40.258833 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.258619    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9792ff\"/\"default-dockercfg-6xggs\""
Sep 18 15:48:40.292948 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.292913    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"]
Sep 18 15:48:40.347694 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.347656    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-dshm\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.347870 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.347720    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-tmp-dir\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.347870 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.347786    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-model-cache\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.347870 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.347812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pgrx\" (UniqueName: \"kubernetes.io/projected/5c8e57a6-40c3-4530-83b9-8b4c502347af-kube-api-access-7pgrx\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.347870 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.347842    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8e57a6-40c3-4530-83b9-8b4c502347af-tls-certs\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.348014 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.347879    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-home\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.448363 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.448328    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-model-cache\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.448363 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.448374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgrx\" (UniqueName: \"kubernetes.io/projected/5c8e57a6-40c3-4530-83b9-8b4c502347af-kube-api-access-7pgrx\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.448604 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.448399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8e57a6-40c3-4530-83b9-8b4c502347af-tls-certs\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.448604 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.448420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-home\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.448604 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.448462    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-dshm\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.448604 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.448484    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-tmp-dir\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.448840 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.448819    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-tmp-dir\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.449097 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.449058    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-home\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.449396 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.449371    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-model-cache\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.451474 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.451454    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8e57a6-40c3-4530-83b9-8b4c502347af-tls-certs\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.451657 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.451639    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-dshm\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.457211 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.457190    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pgrx\" (UniqueName: \"kubernetes.io/projected/5c8e57a6-40c3-4530-83b9-8b4c502347af-kube-api-access-7pgrx\") pod \"simulator-w3-9792ff-kserve-576cffb5-gnbvw\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") " pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.574990 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.574948    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:40.586940 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.586891    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:40.684880 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.683530    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:40.913913 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.913364    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"]
Sep 18 15:48:40.919285 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:48:40.919242    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c8e57a6_40c3_4530_83b9_8b4c502347af.slice/crio-05c633e8043a0442d43e1f99b1fb0165acaeed7fdff817da670738e0099e9cdd WatchSource:0}: Error finding container 05c633e8043a0442d43e1f99b1fb0165acaeed7fdff817da670738e0099e9cdd: Status 404 returned error can't find the container with id 05c633e8043a0442d43e1f99b1fb0165acaeed7fdff817da670738e0099e9cdd
Sep 18 15:48:40.939780 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.939748    2559 patch_prober.go:28] interesting pod/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 18 15:48:40.939933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:40.939815    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc" podUID="df52776c-9cbe-417e-ab2a-4164812cb020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 18 15:48:41.168928 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:41.168794    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:41.240147 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:41.240062    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9" event={"ID":"1492ab45-034b-48c7-a97c-950cfc340a67","Type":"ContainerStarted","Data":"1725f73fc1822296a7e40c3767cab0e70920cc513421ece7180f881f521b1ad9"}
Sep 18 15:48:41.242106 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:41.242081    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw" event={"ID":"5c8e57a6-40c3-4530-83b9-8b4c502347af","Type":"ContainerStarted","Data":"f36d8dbf7c9131657d9923ba66c1cf5e3799bb1b0301be0761e228767c6e1d4a"}
Sep 18 15:48:41.242413 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:41.242115    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw" event={"ID":"5c8e57a6-40c3-4530-83b9-8b4c502347af","Type":"ContainerStarted","Data":"05c633e8043a0442d43e1f99b1fb0165acaeed7fdff817da670738e0099e9cdd"}
Sep 18 15:48:41.256765 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:41.256500    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9" podStartSLOduration=2.121975616 podStartE2EDuration="5.256484674s" totalImagesPullingTime="3.134509058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:41.255971302 +0000 UTC m=+843.817203444" watchObservedRunningTime="2026-09-18 15:48:41.256484674 +0000 UTC m=+843.817716812"
Sep 18 15:48:41.264518 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:41.264446    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:41.280296 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:41.280224    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw" podStartSLOduration=1.280201001 podStartE2EDuration="1.280201001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:41.278805715 +0000 UTC m=+843.840037883" watchObservedRunningTime="2026-09-18 15:48:41.280201001 +0000 UTC m=+843.841433141"
Sep 18 15:48:41.433510 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:41.433479    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:41.939663 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:41.939635    2559 patch_prober.go:28] interesting pod/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 18 15:48:41.939888 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:41.939835    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc" podUID="df52776c-9cbe-417e-ab2a-4164812cb020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 18 15:48:42.148681 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:42.148622    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:42.248619 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:42.248531    2559 generic.go:356] "Generic (PLEG): container finished" podID="1492ab45-034b-48c7-a97c-950cfc340a67" containerID="1725f73fc1822296a7e40c3767cab0e70920cc513421ece7180f881f521b1ad9" exitCode=0
Sep 18 15:48:42.249030 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:42.248999    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9" event={"ID":"1492ab45-034b-48c7-a97c-950cfc340a67","Type":"ContainerDied","Data":"1725f73fc1822296a7e40c3767cab0e70920cc513421ece7180f881f521b1ad9"}
Sep 18 15:48:42.939963 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:42.939933    2559 patch_prober.go:28] interesting pod/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 18 15:48:42.940284 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:42.940263    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc" podUID="df52776c-9cbe-417e-ab2a-4164812cb020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 18 15:48:43.522589 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:43.522527    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9"
Sep 18 15:48:43.610320 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:43.610278    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1492ab45-034b-48c7-a97c-950cfc340a67-maas-api-service-ca\") pod \"1492ab45-034b-48c7-a97c-950cfc340a67\" (UID: \"1492ab45-034b-48c7-a97c-950cfc340a67\") "
Sep 18 15:48:43.611428 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:43.611118    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1492ab45-034b-48c7-a97c-950cfc340a67-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1492ab45-034b-48c7-a97c-950cfc340a67" (UID: "1492ab45-034b-48c7-a97c-950cfc340a67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:48:43.712453 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:43.712420    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1492ab45-034b-48c7-a97c-950cfc340a67-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:43.939312 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:43.939273    2559 patch_prober.go:28] interesting pod/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 18 15:48:43.939482 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:43.939365    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc" podUID="df52776c-9cbe-417e-ab2a-4164812cb020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 18 15:48:44.070092 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:44.070025    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:44.260374 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:44.260285    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9"
Sep 18 15:48:44.261434 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:44.261087    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9" event={"ID":"1492ab45-034b-48c7-a97c-950cfc340a67","Type":"ContainerDied","Data":"bccffef4768e4a58270c53eb7324d1b4252c328f38d669d1394672171ef1db9c"}
Sep 18 15:48:44.261434 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:44.261139    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bccffef4768e4a58270c53eb7324d1b4252c328f38d669d1394672171ef1db9c"
Sep 18 15:48:45.027238 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:45.027205    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:45.030208 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:45.030187    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:48:46.081815 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:46.081777    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:46.959907 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:46.959831    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:47.014612 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.014581    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"]
Sep 18 15:48:47.014807 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.014783    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68" podUID="a82ab850-b1a6-4298-ad8a-4d5a0d1c7513" containerName="payload-processing" containerID="cri-o://0458ffdcac14a15d9fa14aff8d3496530f12c184b039c5b72903e75f16803ec9" gracePeriod=30
Sep 18 15:48:47.039031 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.039003    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"]
Sep 18 15:48:47.039237 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.039213    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5" podUID="54c8096d-2fc9-420b-a6a5-9777962919aa" containerName="payload-pre-processing" containerID="cri-o://c14bc35d070194034f301d7fb9ed205c1c849dc2c08d831578928a642a1a34dd" gracePeriod=30
Sep 18 15:48:47.139567 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.139522    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:47.279652 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.279599    2559 generic.go:356] "Generic (PLEG): container finished" podID="54c8096d-2fc9-420b-a6a5-9777962919aa" containerID="c14bc35d070194034f301d7fb9ed205c1c849dc2c08d831578928a642a1a34dd" exitCode=0
Sep 18 15:48:47.279827 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.279667    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5" event={"ID":"54c8096d-2fc9-420b-a6a5-9777962919aa","Type":"ContainerDied","Data":"c14bc35d070194034f301d7fb9ed205c1c849dc2c08d831578928a642a1a34dd"}
Sep 18 15:48:47.282344 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.282293    2559 generic.go:356] "Generic (PLEG): container finished" podID="a82ab850-b1a6-4298-ad8a-4d5a0d1c7513" containerID="0458ffdcac14a15d9fa14aff8d3496530f12c184b039c5b72903e75f16803ec9" exitCode=0
Sep 18 15:48:47.282344 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.282337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68" event={"ID":"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513","Type":"ContainerDied","Data":"0458ffdcac14a15d9fa14aff8d3496530f12c184b039c5b72903e75f16803ec9"}
Sep 18 15:48:47.294985 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.294946    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:47.341233 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.341207    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"
Sep 18 15:48:47.344121 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.344098    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"
Sep 18 15:48:47.444130 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.444092    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:47.449396 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.449366    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c8096d-2fc9-420b-a6a5-9777962919aa-plugins-config-volume\") pod \"54c8096d-2fc9-420b-a6a5-9777962919aa\" (UID: \"54c8096d-2fc9-420b-a6a5-9777962919aa\") "
Sep 18 15:48:47.449578 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.449461    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmk4n\" (UniqueName: \"kubernetes.io/projected/a82ab850-b1a6-4298-ad8a-4d5a0d1c7513-kube-api-access-zmk4n\") pod \"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513\" (UID: \"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513\") "
Sep 18 15:48:47.449578 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.449505    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4px7\" (UniqueName: \"kubernetes.io/projected/54c8096d-2fc9-420b-a6a5-9777962919aa-kube-api-access-d4px7\") pod \"54c8096d-2fc9-420b-a6a5-9777962919aa\" (UID: \"54c8096d-2fc9-420b-a6a5-9777962919aa\") "
Sep 18 15:48:47.449578 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.449537    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82ab850-b1a6-4298-ad8a-4d5a0d1c7513-plugins-config-volume\") pod \"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513\" (UID: \"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513\") "
Sep 18 15:48:47.449929 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.449849    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c8096d-2fc9-420b-a6a5-9777962919aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54c8096d-2fc9-420b-a6a5-9777962919aa" (UID: "54c8096d-2fc9-420b-a6a5-9777962919aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:48:47.450117 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.450095    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82ab850-b1a6-4298-ad8a-4d5a0d1c7513-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a82ab850-b1a6-4298-ad8a-4d5a0d1c7513" (UID: "a82ab850-b1a6-4298-ad8a-4d5a0d1c7513"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:48:47.451966 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.451935    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c8096d-2fc9-420b-a6a5-9777962919aa-kube-api-access-d4px7" (OuterVolumeSpecName: "kube-api-access-d4px7") pod "54c8096d-2fc9-420b-a6a5-9777962919aa" (UID: "54c8096d-2fc9-420b-a6a5-9777962919aa"). InnerVolumeSpecName "kube-api-access-d4px7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:48:47.452057 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.451957    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82ab850-b1a6-4298-ad8a-4d5a0d1c7513-kube-api-access-zmk4n" (OuterVolumeSpecName: "kube-api-access-zmk4n") pod "a82ab850-b1a6-4298-ad8a-4d5a0d1c7513" (UID: "a82ab850-b1a6-4298-ad8a-4d5a0d1c7513"). InnerVolumeSpecName "kube-api-access-zmk4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:48:47.512765 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.512681    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9"]
Sep 18 15:48:47.518979 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.518950    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8802cb2-q4cn9"]
Sep 18 15:48:47.550751 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.550703    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmk4n\" (UniqueName: \"kubernetes.io/projected/a82ab850-b1a6-4298-ad8a-4d5a0d1c7513-kube-api-access-zmk4n\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:47.550751 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.550747    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4px7\" (UniqueName: \"kubernetes.io/projected/54c8096d-2fc9-420b-a6a5-9777962919aa-kube-api-access-d4px7\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:47.550993 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.550764    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82ab850-b1a6-4298-ad8a-4d5a0d1c7513-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:47.550993 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.550780    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c8096d-2fc9-420b-a6a5-9777962919aa-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:47.579056 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.579017    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:47.671981 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.671941    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"]
Sep 18 15:48:47.672244 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.672213    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z" podUID="1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" containerName="istio-proxy" containerID="cri-o://95971c850cae37c07d9e451a67955076f7027cb47694de8f3328f5a4b8a68105" gracePeriod=30
Sep 18 15:48:47.696776 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.696740    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:47.743970 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:47.743942    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1492ab45-034b-48c7-a97c-950cfc340a67" path="/var/lib/kubelet/pods/1492ab45-034b-48c7-a97c-950cfc340a67/volumes"
Sep 18 15:48:48.013787 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.013766    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:48.156460 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.156377    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-data\") pod \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") "
Sep 18 15:48:48.156460 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.156426    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istiod-ca-cert\") pod \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") "
Sep 18 15:48:48.156460 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.156452    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-workload-socket\") pod \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") "
Sep 18 15:48:48.157018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.156489    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-envoy\") pod \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") "
Sep 18 15:48:48.157018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.156556    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-credential-socket\") pod \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") "
Sep 18 15:48:48.157018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.156614    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg8wt\" (UniqueName: \"kubernetes.io/projected/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-kube-api-access-rg8wt\") pod \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") "
Sep 18 15:48:48.157018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.156638    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-podinfo\") pod \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") "
Sep 18 15:48:48.157018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.156703    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" (UID: "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:48:48.157018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.156719    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-workload-certs\") pod \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") "
Sep 18 15:48:48.157018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.156738    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" (UID: "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:48:48.157018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.156774    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-token\") pod \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\" (UID: \"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81\") "
Sep 18 15:48:48.157018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.156909    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" (UID: "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:48:48.157018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.156942    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" (UID: "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:48:48.157018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.157006    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" (UID: "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:48:48.157510 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.157133    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istiod-ca-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:48.157510 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.157147    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-workload-socket\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:48.157510 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.157157    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-credential-socket\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:48.157510 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.157169    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-workload-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:48.157510 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.157184    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-data\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:48.159149 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.159075    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" (UID: "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:48:48.159281 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.159245    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-kube-api-access-rg8wt" (OuterVolumeSpecName: "kube-api-access-rg8wt") pod "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" (UID: "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81"). InnerVolumeSpecName "kube-api-access-rg8wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:48:48.159378 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.159359    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" (UID: "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 15:48:48.159525 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.159508    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" (UID: "1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:48:48.230340 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.230293    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:48.258343 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.258308    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg8wt\" (UniqueName: \"kubernetes.io/projected/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-kube-api-access-rg8wt\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:48.258343 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.258337    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-podinfo\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:48.258343 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.258348    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-token\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:48.258604 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.258364    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81-istio-envoy\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:48:48.291050 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.291016    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"
Sep 18 15:48:48.291050 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.291026    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5" event={"ID":"54c8096d-2fc9-420b-a6a5-9777962919aa","Type":"ContainerDied","Data":"8bfec7d419b377c85438213a8aa7b1aedfde652670a5b39f4fc721da015ee2de"}
Sep 18 15:48:48.291236 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.291092    2559 scope.go:117] "RemoveContainer" containerID="c14bc35d070194034f301d7fb9ed205c1c849dc2c08d831578928a642a1a34dd"
Sep 18 15:48:48.292466 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.292442    2559 generic.go:356] "Generic (PLEG): container finished" podID="1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" containerID="95971c850cae37c07d9e451a67955076f7027cb47694de8f3328f5a4b8a68105" exitCode=0
Sep 18 15:48:48.292563 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.292495    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"
Sep 18 15:48:48.292563 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.292539    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z" event={"ID":"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81","Type":"ContainerDied","Data":"95971c850cae37c07d9e451a67955076f7027cb47694de8f3328f5a4b8a68105"}
Sep 18 15:48:48.292669 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.292586    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z" event={"ID":"1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81","Type":"ContainerDied","Data":"3d599e40423e477e91210506a338ef64015d8cf850cad53e4f7177c295b5776a"}
Sep 18 15:48:48.294508 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.294484    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68" event={"ID":"a82ab850-b1a6-4298-ad8a-4d5a0d1c7513","Type":"ContainerDied","Data":"d951ee1b151c8976138c9689e73f27221b7ed64b50114333a8958ca964219cbc"}
Sep 18 15:48:48.294508 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.294506    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"
Sep 18 15:48:48.300969 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.300952    2559 scope.go:117] "RemoveContainer" containerID="95971c850cae37c07d9e451a67955076f7027cb47694de8f3328f5a4b8a68105"
Sep 18 15:48:48.309977 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.309953    2559 scope.go:117] "RemoveContainer" containerID="95971c850cae37c07d9e451a67955076f7027cb47694de8f3328f5a4b8a68105"
Sep 18 15:48:48.310314 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:48:48.310294    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95971c850cae37c07d9e451a67955076f7027cb47694de8f3328f5a4b8a68105\": container with ID starting with 95971c850cae37c07d9e451a67955076f7027cb47694de8f3328f5a4b8a68105 not found: ID does not exist" containerID="95971c850cae37c07d9e451a67955076f7027cb47694de8f3328f5a4b8a68105"
Sep 18 15:48:48.310390 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.310324    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95971c850cae37c07d9e451a67955076f7027cb47694de8f3328f5a4b8a68105"} err="failed to get container status \"95971c850cae37c07d9e451a67955076f7027cb47694de8f3328f5a4b8a68105\": rpc error: code = NotFound desc = could not find container \"95971c850cae37c07d9e451a67955076f7027cb47694de8f3328f5a4b8a68105\": container with ID starting with 95971c850cae37c07d9e451a67955076f7027cb47694de8f3328f5a4b8a68105 not found: ID does not exist"
Sep 18 15:48:48.310390 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.310344    2559 scope.go:117] "RemoveContainer" containerID="0458ffdcac14a15d9fa14aff8d3496530f12c184b039c5b72903e75f16803ec9"
Sep 18 15:48:48.361887 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.361833    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"]
Sep 18 15:48:48.386555 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.386517    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a8802cb2-7b4859c94-g4rh5"]
Sep 18 15:48:48.422789 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.422702    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"]
Sep 18 15:48:48.439129 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.439094    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8802cb2-66f9b56d67-drj68"]
Sep 18 15:48:48.495666 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.492002    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"]
Sep 18 15:48:48.527191 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.527156    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:48.538450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.538418    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a8802cb2-openshift-default-76859c6496-cvb6z"]
Sep 18 15:48:48.724050 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.724016    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:48.820560 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.820524    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"]
Sep 18 15:48:48.820963 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.820948    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82ab850-b1a6-4298-ad8a-4d5a0d1c7513" containerName="payload-processing"
Sep 18 15:48:48.820963 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.820964    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82ab850-b1a6-4298-ad8a-4d5a0d1c7513" containerName="payload-processing"
Sep 18 15:48:48.821096 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.820971    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1492ab45-034b-48c7-a97c-950cfc340a67" containerName="revoke-keys"
Sep 18 15:48:48.821096 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.820977    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1492ab45-034b-48c7-a97c-950cfc340a67" containerName="revoke-keys"
Sep 18 15:48:48.821096 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.820990    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c8096d-2fc9-420b-a6a5-9777962919aa" containerName="payload-pre-processing"
Sep 18 15:48:48.821096 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.820999    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c8096d-2fc9-420b-a6a5-9777962919aa" containerName="payload-pre-processing"
Sep 18 15:48:48.821096 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.821021    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" containerName="istio-proxy"
Sep 18 15:48:48.821096 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.821026    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" containerName="istio-proxy"
Sep 18 15:48:48.821096 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.821083    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c8096d-2fc9-420b-a6a5-9777962919aa" containerName="payload-pre-processing"
Sep 18 15:48:48.821096 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.821091    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1492ab45-034b-48c7-a97c-950cfc340a67" containerName="revoke-keys"
Sep 18 15:48:48.821096 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.821097    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a82ab850-b1a6-4298-ad8a-4d5a0d1c7513" containerName="payload-processing"
Sep 18 15:48:48.821361 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.821104    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" containerName="istio-proxy"
Sep 18 15:48:48.824189 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.824165    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:48.828229 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.828209    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-45956207-gw-openshift-default-dockercfg-jknbz\""
Sep 18 15:48:48.843953 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.843926    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"]
Sep 18 15:48:48.872370 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.872331    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:48.964638 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.964600    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-workload-socket\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:48.964845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.964648    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-envoy\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:48.964845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.964709    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e8b085-880f-40f1-8162-28ffed9574dd-istiod-ca-cert\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:48.964845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.964788    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhpxc\" (UniqueName: \"kubernetes.io/projected/c0e8b085-880f-40f1-8162-28ffed9574dd-kube-api-access-jhpxc\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:48.964845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.964816    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-credential-socket\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:48.964845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.964837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-token\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:48.965121 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.964906    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-workload-certs\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:48.965121 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.964929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-podinfo\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:48.965121 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:48.964988    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-data\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.065708 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.065601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-workload-certs\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.065708 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.065638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-podinfo\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.065708 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.065665    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-data\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.065708 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.065711    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-workload-socket\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.066133 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.065733    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-envoy\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.066133 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.065753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e8b085-880f-40f1-8162-28ffed9574dd-istiod-ca-cert\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.066133 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.065784    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhpxc\" (UniqueName: \"kubernetes.io/projected/c0e8b085-880f-40f1-8162-28ffed9574dd-kube-api-access-jhpxc\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.066133 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.065818    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-credential-socket\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.066133 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.065845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-token\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.066390 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.066138    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-workload-certs\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.066390 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.066193    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-data\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.066390 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.066222    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-workload-socket\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.066530 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.066407    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-credential-socket\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.066684 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.066663    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e8b085-880f-40f1-8162-28ffed9574dd-istiod-ca-cert\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.068196 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.068180    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-envoy\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.068413 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.068397    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-podinfo\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.091401 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.091372    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-token\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.093508 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.093484    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhpxc\" (UniqueName: \"kubernetes.io/projected/c0e8b085-880f-40f1-8162-28ffed9574dd-kube-api-access-jhpxc\") pod \"e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") " pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.134294 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.134255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:49.174652 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.174599    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:49.229443 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.229415    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"
Sep 18 15:48:49.416889 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.416796    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"]
Sep 18 15:48:49.420473 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:48:49.420448    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e8b085_880f_40f1_8162_28ffed9574dd.slice/crio-b5c0fa336e1c6ef5db7d92fdb4237a6767a91b3a26f73c270b07c38fb6f6d322 WatchSource:0}: Error finding container b5c0fa336e1c6ef5db7d92fdb4237a6767a91b3a26f73c270b07c38fb6f6d322: Status 404 returned error can't find the container with id b5c0fa336e1c6ef5db7d92fdb4237a6767a91b3a26f73c270b07c38fb6f6d322
Sep 18 15:48:49.422675 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.422649    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:48:49.422730 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.422712    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:48:49.422766 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.422741    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:48:49.691144 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.691113    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:49.744024 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.743995    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81" path="/var/lib/kubelet/pods/1ebe17ad-2f8b-41cc-a090-6af5c5bc6e81/volumes"
Sep 18 15:48:49.744365 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.744353    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c8096d-2fc9-420b-a6a5-9777962919aa" path="/var/lib/kubelet/pods/54c8096d-2fc9-420b-a6a5-9777962919aa/volumes"
Sep 18 15:48:49.744658 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:49.744646    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a82ab850-b1a6-4298-ad8a-4d5a0d1c7513" path="/var/lib/kubelet/pods/a82ab850-b1a6-4298-ad8a-4d5a0d1c7513/volumes"
Sep 18 15:48:50.309263 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:50.309229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79" event={"ID":"c0e8b085-880f-40f1-8162-28ffed9574dd","Type":"ContainerStarted","Data":"3898f4e352972010000461ceba2d2f86a6d974e54968d815f4ef1b44a0d80735"}
Sep 18 15:48:50.309706 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:50.309283    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79" event={"ID":"c0e8b085-880f-40f1-8162-28ffed9574dd","Type":"ContainerStarted","Data":"b5c0fa336e1c6ef5db7d92fdb4237a6767a91b3a26f73c270b07c38fb6f6d322"}
Sep 18 15:48:50.339680 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:50.339627    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79" podStartSLOduration=2.339611119 podStartE2EDuration="2.339611119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:50.338430468 +0000 UTC m=+852.899662640" watchObservedRunningTime="2026-09-18 15:48:50.339611119 +0000 UTC m=+852.900843258"
Sep 18 15:48:50.588077 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:50.587998    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:50.588077 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:50.588046    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:50.600490 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:50.600465    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:50.820082 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:50.820052    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:51.134689 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:51.134642    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:51.140727 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:51.140700    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:51.315035 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:51.315003    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:51.315921 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:51.315900    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:48:51.326280 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:51.326256    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:48:51.853803 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:51.853765    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:51.999597 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:51.999556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:52.164668 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:52.164574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:53.076161 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:53.076106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:54.352481 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:54.352441    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:54.454364 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:54.454332    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:54.816547 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:54.816499    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:54.937238 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:54.937208    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:48:54.977790 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:54.977752    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:55.080311 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.080225    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:55.189365 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.189334    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"]
Sep 18 15:48:55.191798 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.191775    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"
Sep 18 15:48:55.194727 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.194707    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-45956207-dockercfg-xr5c4\""
Sep 18 15:48:55.195100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.195085    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-45956207\""
Sep 18 15:48:55.202744 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.202723    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"]
Sep 18 15:48:55.219100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.219070    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6154601f-c4e5-4849-88b1-bbb1930e3490-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk\" (UID: \"6154601f-c4e5-4849-88b1-bbb1930e3490\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"
Sep 18 15:48:55.219239 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.219124    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk959\" (UniqueName: \"kubernetes.io/projected/6154601f-c4e5-4849-88b1-bbb1930e3490-kube-api-access-dk959\") pod \"payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk\" (UID: \"6154601f-c4e5-4849-88b1-bbb1930e3490\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"
Sep 18 15:48:55.274349 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.274316    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"]
Sep 18 15:48:55.277083 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.277062    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"
Sep 18 15:48:55.292717 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.292692    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"]
Sep 18 15:48:55.319901 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.319865    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk959\" (UniqueName: \"kubernetes.io/projected/6154601f-c4e5-4849-88b1-bbb1930e3490-kube-api-access-dk959\") pod \"payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk\" (UID: \"6154601f-c4e5-4849-88b1-bbb1930e3490\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"
Sep 18 15:48:55.320078 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.319923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6c6805-d412-4a93-bfa9-48da151009f9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-45956207-6869f475db-hrtr7\" (UID: \"bd6c6805-d412-4a93-bfa9-48da151009f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"
Sep 18 15:48:55.320078 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.319970    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssz8j\" (UniqueName: \"kubernetes.io/projected/bd6c6805-d412-4a93-bfa9-48da151009f9-kube-api-access-ssz8j\") pod \"payload-processing-e2e-migrate-45956207-6869f475db-hrtr7\" (UID: \"bd6c6805-d412-4a93-bfa9-48da151009f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"
Sep 18 15:48:55.320078 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.320045    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6154601f-c4e5-4849-88b1-bbb1930e3490-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk\" (UID: \"6154601f-c4e5-4849-88b1-bbb1930e3490\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"
Sep 18 15:48:55.320734 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.320714    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6154601f-c4e5-4849-88b1-bbb1930e3490-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk\" (UID: \"6154601f-c4e5-4849-88b1-bbb1930e3490\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"
Sep 18 15:48:55.333703 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.333595    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk959\" (UniqueName: \"kubernetes.io/projected/6154601f-c4e5-4849-88b1-bbb1930e3490-kube-api-access-dk959\") pod \"payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk\" (UID: \"6154601f-c4e5-4849-88b1-bbb1930e3490\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"
Sep 18 15:48:55.420464 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.420426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6c6805-d412-4a93-bfa9-48da151009f9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-45956207-6869f475db-hrtr7\" (UID: \"bd6c6805-d412-4a93-bfa9-48da151009f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"
Sep 18 15:48:55.420844 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.420487    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssz8j\" (UniqueName: \"kubernetes.io/projected/bd6c6805-d412-4a93-bfa9-48da151009f9-kube-api-access-ssz8j\") pod \"payload-processing-e2e-migrate-45956207-6869f475db-hrtr7\" (UID: \"bd6c6805-d412-4a93-bfa9-48da151009f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"
Sep 18 15:48:55.421130 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.421111    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6c6805-d412-4a93-bfa9-48da151009f9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-45956207-6869f475db-hrtr7\" (UID: \"bd6c6805-d412-4a93-bfa9-48da151009f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"
Sep 18 15:48:55.428810 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.428781    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssz8j\" (UniqueName: \"kubernetes.io/projected/bd6c6805-d412-4a93-bfa9-48da151009f9-kube-api-access-ssz8j\") pod \"payload-processing-e2e-migrate-45956207-6869f475db-hrtr7\" (UID: \"bd6c6805-d412-4a93-bfa9-48da151009f9\") " pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"
Sep 18 15:48:55.503617 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.503592    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"
Sep 18 15:48:55.556985 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.556946    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:55.587533 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.587462    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"
Sep 18 15:48:55.781278 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.781254    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"]
Sep 18 15:48:55.782794 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:48:55.782765    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6154601f_c4e5_4849_88b1_bbb1930e3490.slice/crio-ec3288c88c44fdb8f28e096b86a9b7800d103242368b54c74c0727ba8f303327 WatchSource:0}: Error finding container ec3288c88c44fdb8f28e096b86a9b7800d103242368b54c74c0727ba8f303327: Status 404 returned error can't find the container with id ec3288c88c44fdb8f28e096b86a9b7800d103242368b54c74c0727ba8f303327
Sep 18 15:48:55.865773 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:55.865690    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"]
Sep 18 15:48:55.869392 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:48:55.869366    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6c6805_d412_4a93_bfa9_48da151009f9.slice/crio-5f2caa4dabe00319f356a10a2d98430673a1834120745d1c0f5a708001b050a6 WatchSource:0}: Error finding container 5f2caa4dabe00319f356a10a2d98430673a1834120745d1c0f5a708001b050a6: Status 404 returned error can't find the container with id 5f2caa4dabe00319f356a10a2d98430673a1834120745d1c0f5a708001b050a6
Sep 18 15:48:56.343375 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:56.343341    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk" event={"ID":"6154601f-c4e5-4849-88b1-bbb1930e3490","Type":"ContainerStarted","Data":"21176cf2d0a198dd9f861406a3ca5efbedf3ac8ce10038bcbafdff404de9304c"}
Sep 18 15:48:56.343542 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:56.343401    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk" event={"ID":"6154601f-c4e5-4849-88b1-bbb1930e3490","Type":"ContainerStarted","Data":"ec3288c88c44fdb8f28e096b86a9b7800d103242368b54c74c0727ba8f303327"}
Sep 18 15:48:56.343542 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:56.343452    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"
Sep 18 15:48:56.345145 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:56.345118    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7" event={"ID":"bd6c6805-d412-4a93-bfa9-48da151009f9","Type":"ContainerStarted","Data":"95a677e18cdede44016557336d277947ca11970ec54251525ae11c0aa3c9a0f9"}
Sep 18 15:48:56.345251 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:56.345165    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7" event={"ID":"bd6c6805-d412-4a93-bfa9-48da151009f9","Type":"ContainerStarted","Data":"5f2caa4dabe00319f356a10a2d98430673a1834120745d1c0f5a708001b050a6"}
Sep 18 15:48:56.345251 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:56.345202    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"
Sep 18 15:48:56.374459 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:56.374412    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk" podStartSLOduration=1.374398081 podStartE2EDuration="1.374398081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:56.371969971 +0000 UTC m=+858.933202109" watchObservedRunningTime="2026-09-18 15:48:56.374398081 +0000 UTC m=+858.935630221"
Sep 18 15:48:56.396700 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:56.396651    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7" podStartSLOduration=1.396636335 podStartE2EDuration="1.396636335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:56.396223186 +0000 UTC m=+858.957455326" watchObservedRunningTime="2026-09-18 15:48:56.396636335 +0000 UTC m=+858.957868474"
Sep 18 15:48:56.627056 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:56.626969    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:57.486430 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:57.486386    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:57.588293 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:57.588260    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:58.302384 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:58.302354    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:58.540609 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:58.540576    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:58.644658 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:58.644569    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:48:59.230113 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:48:59.230081    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:00.018674 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:00.018639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:04.153975 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:04.153945    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:07.352360 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:07.352332    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"
Sep 18 15:49:07.352825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:07.352644    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"
Sep 18 15:49:12.241330 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:12.241291    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:12.889121 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:12.889088    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:49:14.444741 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:14.444707    2559 generic.go:356] "Generic (PLEG): container finished" podID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerID="3898f4e352972010000461ceba2d2f86a6d974e54968d815f4ef1b44a0d80735" exitCode=0
Sep 18 15:49:14.445307 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:14.444774    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79" event={"ID":"c0e8b085-880f-40f1-8162-28ffed9574dd","Type":"ContainerDied","Data":"3898f4e352972010000461ceba2d2f86a6d974e54968d815f4ef1b44a0d80735"}
Sep 18 15:49:14.445307 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:14.445293    2559 scope.go:117] "RemoveContainer" containerID="3898f4e352972010000461ceba2d2f86a6d974e54968d815f4ef1b44a0d80735"
Sep 18 15:49:14.446492 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:14.446456    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:49:14.446576 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:14.446528    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:49:14.446576 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:14.446559    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:49:15.452060 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:15.452024    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79" event={"ID":"c0e8b085-880f-40f1-8162-28ffed9574dd","Type":"ContainerStarted","Data":"45358292a62d46518c76475d4ac25af22a37af0302cf33c28c4a394c35a7de72"}
Sep 18 15:49:16.134553 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.134516    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:49:16.134805 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.134787    2559 patch_prober.go:28] interesting pod/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 18 15:49:16.134878 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.134829    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79" podUID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 18 15:49:16.641384 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.641351    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"]
Sep 18 15:49:16.644194 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.644172    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.648634 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.648612    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a1f773\"/\"default-dockercfg-77n64\""
Sep 18 15:49:16.648739 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.648642    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a1f773\"/\"openshift-service-ca.crt\""
Sep 18 15:49:16.648739 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.648645    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a1f773\"/\"kube-root-ca.crt\""
Sep 18 15:49:16.648825 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.648612    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a1f773\"/\"test-model-a1f773-kserve-self-signed-certs\""
Sep 18 15:49:16.658231 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.658210    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"]
Sep 18 15:49:16.699160 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.699017    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:16.704770 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.704745    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-model-cache\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.704923 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.704779    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-tmp-dir\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.704923 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.704803    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-home\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.705031 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.704932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-dshm\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.705031 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.704962    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ghfj\" (UniqueName: \"kubernetes.io/projected/75e9b9a9-1a1e-4428-8653-a994ac150d09-kube-api-access-7ghfj\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.705127 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.705045    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e9b9a9-1a1e-4428-8653-a994ac150d09-tls-certs\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.806529 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.806436    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-model-cache\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.806529 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.806480    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-tmp-dir\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.806529 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.806504    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-home\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.806802 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.806555    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-dshm\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.806802 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.806571    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ghfj\" (UniqueName: \"kubernetes.io/projected/75e9b9a9-1a1e-4428-8653-a994ac150d09-kube-api-access-7ghfj\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.806802 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.806633    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e9b9a9-1a1e-4428-8653-a994ac150d09-tls-certs\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.807004 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.806916    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-model-cache\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.807004 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.806953    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-tmp-dir\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.807004 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.806994    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-home\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.809005 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.808983    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-dshm\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.809144 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.809129    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e9b9a9-1a1e-4428-8653-a994ac150d09-tls-certs\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.820211 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.820191    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ghfj\" (UniqueName: \"kubernetes.io/projected/75e9b9a9-1a1e-4428-8653-a994ac150d09-kube-api-access-7ghfj\") pod \"test-model-a1f773-kserve-74ddf875b7-wp86g\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") " pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.843420 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.843395    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc"]
Sep 18 15:49:16.846088 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.846073    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc"
Sep 18 15:49:16.850452 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.850426    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 15:49:16.850771 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.850757    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 15:49:16.873924 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.873889    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc"]
Sep 18 15:49:16.907296 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.907251    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a7fe92-c0b4-4142-8840-678b8cff4e72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-45956207-8t9fc\" (UID: \"c0a7fe92-c0b4-4142-8840-678b8cff4e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc"
Sep 18 15:49:16.980287 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.980256    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:16.980724 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:16.980690    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:17.008539 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:17.008505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a7fe92-c0b4-4142-8840-678b8cff4e72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-45956207-8t9fc\" (UID: \"c0a7fe92-c0b4-4142-8840-678b8cff4e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc"
Sep 18 15:49:17.009150 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:17.009130    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a7fe92-c0b4-4142-8840-678b8cff4e72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-45956207-8t9fc\" (UID: \"c0a7fe92-c0b4-4142-8840-678b8cff4e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc"
Sep 18 15:49:17.135332 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:17.135233    2559 patch_prober.go:28] interesting pod/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 18 15:49:17.135332 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:17.135311    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79" podUID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 18 15:49:17.156512 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:17.156479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc"
Sep 18 15:49:17.282779 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:17.282737    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"]
Sep 18 15:49:17.289832 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:49:17.289791    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75e9b9a9_1a1e_4428_8653_a994ac150d09.slice/crio-67522ce9f54cd38073678f5a1a0a8a6de67235e1af1151fa08b0bad8382fef94 WatchSource:0}: Error finding container 67522ce9f54cd38073678f5a1a0a8a6de67235e1af1151fa08b0bad8382fef94: Status 404 returned error can't find the container with id 67522ce9f54cd38073678f5a1a0a8a6de67235e1af1151fa08b0bad8382fef94
Sep 18 15:49:17.446832 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:17.446805    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc"]
Sep 18 15:49:17.448503 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:49:17.448473    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a7fe92_c0b4_4142_8840_678b8cff4e72.slice/crio-58dd94e88f28e21e4eb72a7ab189be838edc50470d8045b69684687d73bbdbd0 WatchSource:0}: Error finding container 58dd94e88f28e21e4eb72a7ab189be838edc50470d8045b69684687d73bbdbd0: Status 404 returned error can't find the container with id 58dd94e88f28e21e4eb72a7ab189be838edc50470d8045b69684687d73bbdbd0
Sep 18 15:49:17.470849 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:17.470813    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc" event={"ID":"c0a7fe92-c0b4-4142-8840-678b8cff4e72","Type":"ContainerStarted","Data":"58dd94e88f28e21e4eb72a7ab189be838edc50470d8045b69684687d73bbdbd0"}
Sep 18 15:49:17.479461 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:17.479429    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g" event={"ID":"75e9b9a9-1a1e-4428-8653-a994ac150d09","Type":"ContainerStarted","Data":"6cbe22ed024b1f1deddd81ff6e67ae5a8ff5620d1310b25c1b153a84eb9c5e85"}
Sep 18 15:49:17.479609 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:17.479476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g" event={"ID":"75e9b9a9-1a1e-4428-8653-a994ac150d09","Type":"ContainerStarted","Data":"67522ce9f54cd38073678f5a1a0a8a6de67235e1af1151fa08b0bad8382fef94"}
Sep 18 15:49:17.517469 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:17.517424    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g" podStartSLOduration=1.5174103570000002 podStartE2EDuration="1.517410357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:49:17.516096665 +0000 UTC m=+880.077328815" watchObservedRunningTime="2026-09-18 15:49:17.517410357 +0000 UTC m=+880.078642495"
Sep 18 15:49:17.803331 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:17.803296    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:18.135546 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:18.135463    2559 patch_prober.go:28] interesting pod/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 18 15:49:18.135546 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:18.135515    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79" podUID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 18 15:49:18.488523 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:18.488486    2559 generic.go:356] "Generic (PLEG): container finished" podID="c0a7fe92-c0b4-4142-8840-678b8cff4e72" containerID="bfff34b8dc2366dfe30f06e1cba60e32de20a2a511933553ec59e18ba13a1290" exitCode=0
Sep 18 15:49:18.488711 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:18.488559    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc" event={"ID":"c0a7fe92-c0b4-4142-8840-678b8cff4e72","Type":"ContainerDied","Data":"bfff34b8dc2366dfe30f06e1cba60e32de20a2a511933553ec59e18ba13a1290"}
Sep 18 15:49:18.807694 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:18.807655    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:19.135129 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:19.135040    2559 patch_prober.go:28] interesting pod/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 18 15:49:19.135129 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:19.135100    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79" podUID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 18 15:49:19.606417 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:19.606377    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:19.647134 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:19.647109    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc"
Sep 18 15:49:19.735497 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:19.735468    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a7fe92-c0b4-4142-8840-678b8cff4e72-maas-api-service-ca\") pod \"c0a7fe92-c0b4-4142-8840-678b8cff4e72\" (UID: \"c0a7fe92-c0b4-4142-8840-678b8cff4e72\") "
Sep 18 15:49:19.735813 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:19.735788    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a7fe92-c0b4-4142-8840-678b8cff4e72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0a7fe92-c0b4-4142-8840-678b8cff4e72" (UID: "c0a7fe92-c0b4-4142-8840-678b8cff4e72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:49:19.836833 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:19.836797    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a7fe92-c0b4-4142-8840-678b8cff4e72-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:20.135454 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:20.135358    2559 patch_prober.go:28] interesting pod/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 18 15:49:20.135454 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:20.135418    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79" podUID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 18 15:49:20.380491 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:20.380452    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:20.500702 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:20.500677    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc"
Sep 18 15:49:20.500702 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:20.500695    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc" event={"ID":"c0a7fe92-c0b4-4142-8840-678b8cff4e72","Type":"ContainerDied","Data":"58dd94e88f28e21e4eb72a7ab189be838edc50470d8045b69684687d73bbdbd0"}
Sep 18 15:49:20.500951 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:20.500735    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58dd94e88f28e21e4eb72a7ab189be838edc50470d8045b69684687d73bbdbd0"
Sep 18 15:49:21.139218 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:21.139181    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:49:21.139657 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:21.139474    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:49:21.140328 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:21.140277    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:49:21.279792 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:21.279759    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:22.208905 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:22.208864    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:23.311159 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:23.311122    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:23.905221 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:23.905178    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:23.948288 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:23.948258    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"]
Sep 18 15:49:23.948516 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:23.948490    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7" podUID="bd6c6805-d412-4a93-bfa9-48da151009f9" containerName="payload-processing" containerID="cri-o://95a677e18cdede44016557336d277947ca11970ec54251525ae11c0aa3c9a0f9" gracePeriod=30
Sep 18 15:49:23.958164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:23.957830    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"]
Sep 18 15:49:23.958665 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:23.958619    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk" podUID="6154601f-c4e5-4849-88b1-bbb1930e3490" containerName="payload-pre-processing" containerID="cri-o://21176cf2d0a198dd9f861406a3ca5efbedf3ac8ce10038bcbafdff404de9304c" gracePeriod=30
Sep 18 15:49:23.999197 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:23.999168    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:24.066256 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.066230    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:24.165515 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.165485    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:24.167961 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.167930    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc"]
Sep 18 15:49:24.169701 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.169677    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45956207-8t9fc"]
Sep 18 15:49:24.209592 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.209571    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"
Sep 18 15:49:24.229810 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.229739    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:24.236865 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.236828    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"
Sep 18 15:49:24.275156 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.275126    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6c6805-d412-4a93-bfa9-48da151009f9-plugins-config-volume\") pod \"bd6c6805-d412-4a93-bfa9-48da151009f9\" (UID: \"bd6c6805-d412-4a93-bfa9-48da151009f9\") "
Sep 18 15:49:24.275340 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.275245    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssz8j\" (UniqueName: \"kubernetes.io/projected/bd6c6805-d412-4a93-bfa9-48da151009f9-kube-api-access-ssz8j\") pod \"bd6c6805-d412-4a93-bfa9-48da151009f9\" (UID: \"bd6c6805-d412-4a93-bfa9-48da151009f9\") "
Sep 18 15:49:24.275633 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.275605    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6c6805-d412-4a93-bfa9-48da151009f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd6c6805-d412-4a93-bfa9-48da151009f9" (UID: "bd6c6805-d412-4a93-bfa9-48da151009f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:49:24.278908 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.278879    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6c6805-d412-4a93-bfa9-48da151009f9-kube-api-access-ssz8j" (OuterVolumeSpecName: "kube-api-access-ssz8j") pod "bd6c6805-d412-4a93-bfa9-48da151009f9" (UID: "bd6c6805-d412-4a93-bfa9-48da151009f9"). InnerVolumeSpecName "kube-api-access-ssz8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:49:24.300918 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.300890    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"]
Sep 18 15:49:24.301125 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.301100    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79" podUID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerName="istio-proxy" containerID="cri-o://45358292a62d46518c76475d4ac25af22a37af0302cf33c28c4a394c35a7de72" gracePeriod=30
Sep 18 15:49:24.323312 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.323286    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:24.375924 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.375891    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6154601f-c4e5-4849-88b1-bbb1930e3490-plugins-config-volume\") pod \"6154601f-c4e5-4849-88b1-bbb1930e3490\" (UID: \"6154601f-c4e5-4849-88b1-bbb1930e3490\") "
Sep 18 15:49:24.376117 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.375980    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk959\" (UniqueName: \"kubernetes.io/projected/6154601f-c4e5-4849-88b1-bbb1930e3490-kube-api-access-dk959\") pod \"6154601f-c4e5-4849-88b1-bbb1930e3490\" (UID: \"6154601f-c4e5-4849-88b1-bbb1930e3490\") "
Sep 18 15:49:24.376236 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.376219    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssz8j\" (UniqueName: \"kubernetes.io/projected/bd6c6805-d412-4a93-bfa9-48da151009f9-kube-api-access-ssz8j\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:24.376297 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.376242    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6c6805-d412-4a93-bfa9-48da151009f9-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:24.376297 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.376234    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6154601f-c4e5-4849-88b1-bbb1930e3490-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6154601f-c4e5-4849-88b1-bbb1930e3490" (UID: "6154601f-c4e5-4849-88b1-bbb1930e3490"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:49:24.377988 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.377962    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6154601f-c4e5-4849-88b1-bbb1930e3490-kube-api-access-dk959" (OuterVolumeSpecName: "kube-api-access-dk959") pod "6154601f-c4e5-4849-88b1-bbb1930e3490" (UID: "6154601f-c4e5-4849-88b1-bbb1930e3490"). InnerVolumeSpecName "kube-api-access-dk959". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:49:24.477548 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.477512    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6154601f-c4e5-4849-88b1-bbb1930e3490-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:24.477548 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.477542    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk959\" (UniqueName: \"kubernetes.io/projected/6154601f-c4e5-4849-88b1-bbb1930e3490-kube-api-access-dk959\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:24.521680 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.521642    2559 generic.go:356] "Generic (PLEG): container finished" podID="6154601f-c4e5-4849-88b1-bbb1930e3490" containerID="21176cf2d0a198dd9f861406a3ca5efbedf3ac8ce10038bcbafdff404de9304c" exitCode=0
Sep 18 15:49:24.521833 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.521701    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"
Sep 18 15:49:24.521833 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.521734    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk" event={"ID":"6154601f-c4e5-4849-88b1-bbb1930e3490","Type":"ContainerDied","Data":"21176cf2d0a198dd9f861406a3ca5efbedf3ac8ce10038bcbafdff404de9304c"}
Sep 18 15:49:24.521833 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.521818    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk" event={"ID":"6154601f-c4e5-4849-88b1-bbb1930e3490","Type":"ContainerDied","Data":"ec3288c88c44fdb8f28e096b86a9b7800d103242368b54c74c0727ba8f303327"}
Sep 18 15:49:24.522015 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.521875    2559 scope.go:117] "RemoveContainer" containerID="21176cf2d0a198dd9f861406a3ca5efbedf3ac8ce10038bcbafdff404de9304c"
Sep 18 15:49:24.523185 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.523160    2559 generic.go:356] "Generic (PLEG): container finished" podID="bd6c6805-d412-4a93-bfa9-48da151009f9" containerID="95a677e18cdede44016557336d277947ca11970ec54251525ae11c0aa3c9a0f9" exitCode=0
Sep 18 15:49:24.523256 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.523218    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"
Sep 18 15:49:24.523256 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.523219    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7" event={"ID":"bd6c6805-d412-4a93-bfa9-48da151009f9","Type":"ContainerDied","Data":"95a677e18cdede44016557336d277947ca11970ec54251525ae11c0aa3c9a0f9"}
Sep 18 15:49:24.523353 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.523289    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7" event={"ID":"bd6c6805-d412-4a93-bfa9-48da151009f9","Type":"ContainerDied","Data":"5f2caa4dabe00319f356a10a2d98430673a1834120745d1c0f5a708001b050a6"}
Sep 18 15:49:24.533800 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.533708    2559 scope.go:117] "RemoveContainer" containerID="21176cf2d0a198dd9f861406a3ca5efbedf3ac8ce10038bcbafdff404de9304c"
Sep 18 15:49:24.534068 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:49:24.534029    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21176cf2d0a198dd9f861406a3ca5efbedf3ac8ce10038bcbafdff404de9304c\": container with ID starting with 21176cf2d0a198dd9f861406a3ca5efbedf3ac8ce10038bcbafdff404de9304c not found: ID does not exist" containerID="21176cf2d0a198dd9f861406a3ca5efbedf3ac8ce10038bcbafdff404de9304c"
Sep 18 15:49:24.534144 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.534074    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21176cf2d0a198dd9f861406a3ca5efbedf3ac8ce10038bcbafdff404de9304c"} err="failed to get container status \"21176cf2d0a198dd9f861406a3ca5efbedf3ac8ce10038bcbafdff404de9304c\": rpc error: code = NotFound desc = could not find container \"21176cf2d0a198dd9f861406a3ca5efbedf3ac8ce10038bcbafdff404de9304c\": container with ID starting with 21176cf2d0a198dd9f861406a3ca5efbedf3ac8ce10038bcbafdff404de9304c not found: ID does not exist"
Sep 18 15:49:24.534144 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.534091    2559 scope.go:117] "RemoveContainer" containerID="95a677e18cdede44016557336d277947ca11970ec54251525ae11c0aa3c9a0f9"
Sep 18 15:49:24.541870 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.541840    2559 scope.go:117] "RemoveContainer" containerID="95a677e18cdede44016557336d277947ca11970ec54251525ae11c0aa3c9a0f9"
Sep 18 15:49:24.542127 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:49:24.542109    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a677e18cdede44016557336d277947ca11970ec54251525ae11c0aa3c9a0f9\": container with ID starting with 95a677e18cdede44016557336d277947ca11970ec54251525ae11c0aa3c9a0f9 not found: ID does not exist" containerID="95a677e18cdede44016557336d277947ca11970ec54251525ae11c0aa3c9a0f9"
Sep 18 15:49:24.542176 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.542134    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a677e18cdede44016557336d277947ca11970ec54251525ae11c0aa3c9a0f9"} err="failed to get container status \"95a677e18cdede44016557336d277947ca11970ec54251525ae11c0aa3c9a0f9\": rpc error: code = NotFound desc = could not find container \"95a677e18cdede44016557336d277947ca11970ec54251525ae11c0aa3c9a0f9\": container with ID starting with 95a677e18cdede44016557336d277947ca11970ec54251525ae11c0aa3c9a0f9 not found: ID does not exist"
Sep 18 15:49:24.553172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.553152    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"]
Sep 18 15:49:24.557034 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.557012    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-45956207-6869f475db-hrtr7"]
Sep 18 15:49:24.580118 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.580099    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"]
Sep 18 15:49:24.586177 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.586157    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-45956207-59cb4c88b6-t6vvk"]
Sep 18 15:49:24.612488 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.612461    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:24.766981 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:24.766897    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:25.063638 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.063556    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"]
Sep 18 15:49:25.064171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.064151    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a7fe92-c0b4-4142-8840-678b8cff4e72" containerName="revoke-keys"
Sep 18 15:49:25.064223 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.064175    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a7fe92-c0b4-4142-8840-678b8cff4e72" containerName="revoke-keys"
Sep 18 15:49:25.064223 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.064193    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd6c6805-d412-4a93-bfa9-48da151009f9" containerName="payload-processing"
Sep 18 15:49:25.064223 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.064202    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd6c6805-d412-4a93-bfa9-48da151009f9" containerName="payload-processing"
Sep 18 15:49:25.064223 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.064219    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6154601f-c4e5-4849-88b1-bbb1930e3490" containerName="payload-pre-processing"
Sep 18 15:49:25.064340 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.064228    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6154601f-c4e5-4849-88b1-bbb1930e3490" containerName="payload-pre-processing"
Sep 18 15:49:25.064340 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.064328    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0a7fe92-c0b4-4142-8840-678b8cff4e72" containerName="revoke-keys"
Sep 18 15:49:25.064411 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.064341    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd6c6805-d412-4a93-bfa9-48da151009f9" containerName="payload-processing"
Sep 18 15:49:25.064411 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.064353    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6154601f-c4e5-4849-88b1-bbb1930e3490" containerName="payload-pre-processing"
Sep 18 15:49:25.066519 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.066501    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.069122 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.069102    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d9680198-openshift-default-dockercfg-vw7qk\""
Sep 18 15:49:25.079564 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.079544    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"]
Sep 18 15:49:25.093349 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.093319    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:25.184351 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.184316    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istiod-ca-cert\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.184538 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.184371    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-envoy\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.184538 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.184397    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4vqv\" (UniqueName: \"kubernetes.io/projected/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-kube-api-access-q4vqv\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.184538 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.184432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-workload-socket\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.184538 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.184467    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-workload-certs\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.184538 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.184490    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-credential-socket\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.184761 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.184584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-token\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.184761 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.184633    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-data\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.184761 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.184674    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-podinfo\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.285868 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.285820    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-workload-certs\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.286076 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.285887    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-credential-socket\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.286076 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.285921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-token\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.286076 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.285955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-data\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.286076 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.285982    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-podinfo\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.286076 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.286074    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istiod-ca-cert\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.286362 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.286130    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-envoy\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.286458 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.286430    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4vqv\" (UniqueName: \"kubernetes.io/projected/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-kube-api-access-q4vqv\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.286576 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.286508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-workload-socket\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.286649 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.286599    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-credential-socket\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.286649 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.286631    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-data\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.286764 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.286737    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-workload-certs\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.287053 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.287034    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istiod-ca-cert\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.287282 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.287265    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-workload-socket\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.288428 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.288404    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-envoy\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.288525 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.288474    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-podinfo\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.295605 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.295583    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-token\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.296234 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.296209    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4vqv\" (UniqueName: \"kubernetes.io/projected/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-kube-api-access-q4vqv\") pod \"e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") " pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.376562 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.376483    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:25.456976 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.456870    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:25.584893 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.584844    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:25.654269 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.654238    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"]
Sep 18 15:49:25.654624 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:49:25.654603    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e78b34_9bf9_4bd8_be20_c6a3fbf4a0f4.slice/crio-b6012be1c71ad709f64f6821a6755f79c15078f1c0b1ddb2104d694ac47b86a3 WatchSource:0}: Error finding container b6012be1c71ad709f64f6821a6755f79c15078f1c0b1ddb2104d694ac47b86a3: Status 404 returned error can't find the container with id b6012be1c71ad709f64f6821a6755f79c15078f1c0b1ddb2104d694ac47b86a3
Sep 18 15:49:25.656897 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.656847    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:49:25.657000 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.656942    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:49:25.657000 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.656985    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:49:25.746346 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.746309    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6154601f-c4e5-4849-88b1-bbb1930e3490" path="/var/lib/kubelet/pods/6154601f-c4e5-4849-88b1-bbb1930e3490/volumes"
Sep 18 15:49:25.746890 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.746848    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd6c6805-d412-4a93-bfa9-48da151009f9" path="/var/lib/kubelet/pods/bd6c6805-d412-4a93-bfa9-48da151009f9/volumes"
Sep 18 15:49:25.747327 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:25.747305    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0a7fe92-c0b4-4142-8840-678b8cff4e72" path="/var/lib/kubelet/pods/c0a7fe92-c0b4-4142-8840-678b8cff4e72/volumes"
Sep 18 15:49:26.194744 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:26.194708    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:26.538557 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:26.538468    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" event={"ID":"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4","Type":"ContainerStarted","Data":"d8720799028386081609a02e9b409643a6564f8fe3380ce2630e4f10053ae54a"}
Sep 18 15:49:26.538557 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:26.538519    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" event={"ID":"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4","Type":"ContainerStarted","Data":"b6012be1c71ad709f64f6821a6755f79c15078f1c0b1ddb2104d694ac47b86a3"}
Sep 18 15:49:26.562550 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:26.562484    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" podStartSLOduration=1.56246256 podStartE2EDuration="1.56246256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:49:26.562334186 +0000 UTC m=+889.123566326" watchObservedRunningTime="2026-09-18 15:49:26.56246256 +0000 UTC m=+889.123694701"
Sep 18 15:49:26.981526 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:26.981496    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:26.981526 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:26.981537    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:26.994367 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:26.994341    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:27.015058 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:27.015028    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:27.377584 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:27.377492    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:27.383157 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:27.383131    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:27.543333 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:27.543304    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:27.544200 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:27.544182    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:27.554356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:27.554335    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:49:27.773010 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:27.772976    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:27.922818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:27.922782    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:28.057161 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:28.057074    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:28.577286 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:28.577244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:29.384922 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.384884    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:29.555565 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.555537    2559 generic.go:356] "Generic (PLEG): container finished" podID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerID="45358292a62d46518c76475d4ac25af22a37af0302cf33c28c4a394c35a7de72" exitCode=0
Sep 18 15:49:29.555721 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.555607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79" event={"ID":"c0e8b085-880f-40f1-8162-28ffed9574dd","Type":"ContainerDied","Data":"45358292a62d46518c76475d4ac25af22a37af0302cf33c28c4a394c35a7de72"}
Sep 18 15:49:29.555721 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.555659    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79" event={"ID":"c0e8b085-880f-40f1-8162-28ffed9574dd","Type":"ContainerDied","Data":"b5c0fa336e1c6ef5db7d92fdb4237a6767a91b3a26f73c270b07c38fb6f6d322"}
Sep 18 15:49:29.555721 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.555676    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5c0fa336e1c6ef5db7d92fdb4237a6767a91b3a26f73c270b07c38fb6f6d322"
Sep 18 15:49:29.555721 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.555689    2559 scope.go:117] "RemoveContainer" containerID="3898f4e352972010000461ceba2d2f86a6d974e54968d815f4ef1b44a0d80735"
Sep 18 15:49:29.555923 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.555614    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:49:29.628435 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.628393    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-workload-certs\") pod \"c0e8b085-880f-40f1-8162-28ffed9574dd\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") "
Sep 18 15:49:29.628934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.628461    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-podinfo\") pod \"c0e8b085-880f-40f1-8162-28ffed9574dd\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") "
Sep 18 15:49:29.628934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.628486    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-token\") pod \"c0e8b085-880f-40f1-8162-28ffed9574dd\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") "
Sep 18 15:49:29.628934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.628521    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-workload-socket\") pod \"c0e8b085-880f-40f1-8162-28ffed9574dd\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") "
Sep 18 15:49:29.628934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.628559    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhpxc\" (UniqueName: \"kubernetes.io/projected/c0e8b085-880f-40f1-8162-28ffed9574dd-kube-api-access-jhpxc\") pod \"c0e8b085-880f-40f1-8162-28ffed9574dd\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") "
Sep 18 15:49:29.628934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.628577    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-data\") pod \"c0e8b085-880f-40f1-8162-28ffed9574dd\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") "
Sep 18 15:49:29.628934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.628624    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0e8b085-880f-40f1-8162-28ffed9574dd" (UID: "c0e8b085-880f-40f1-8162-28ffed9574dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:49:29.628934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.628638    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e8b085-880f-40f1-8162-28ffed9574dd-istiod-ca-cert\") pod \"c0e8b085-880f-40f1-8162-28ffed9574dd\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") "
Sep 18 15:49:29.628934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.628678    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-envoy\") pod \"c0e8b085-880f-40f1-8162-28ffed9574dd\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") "
Sep 18 15:49:29.628934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.628713    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-credential-socket\") pod \"c0e8b085-880f-40f1-8162-28ffed9574dd\" (UID: \"c0e8b085-880f-40f1-8162-28ffed9574dd\") "
Sep 18 15:49:29.628934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.628796    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0e8b085-880f-40f1-8162-28ffed9574dd" (UID: "c0e8b085-880f-40f1-8162-28ffed9574dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:49:29.628934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.628922    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0e8b085-880f-40f1-8162-28ffed9574dd" (UID: "c0e8b085-880f-40f1-8162-28ffed9574dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:49:29.629540 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.628999    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e8b085-880f-40f1-8162-28ffed9574dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0e8b085-880f-40f1-8162-28ffed9574dd" (UID: "c0e8b085-880f-40f1-8162-28ffed9574dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:49:29.629540 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.629213    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0e8b085-880f-40f1-8162-28ffed9574dd" (UID: "c0e8b085-880f-40f1-8162-28ffed9574dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:49:29.629540 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.629512    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-workload-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:29.629540 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.629532    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-workload-socket\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:29.629744 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.629545    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-data\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:29.629744 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.629561    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e8b085-880f-40f1-8162-28ffed9574dd-istiod-ca-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:29.629744 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.629573    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-credential-socket\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:29.631004 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.630977    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e8b085-880f-40f1-8162-28ffed9574dd-kube-api-access-jhpxc" (OuterVolumeSpecName: "kube-api-access-jhpxc") pod "c0e8b085-880f-40f1-8162-28ffed9574dd" (UID: "c0e8b085-880f-40f1-8162-28ffed9574dd"). InnerVolumeSpecName "kube-api-access-jhpxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:49:29.631123 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.631028    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0e8b085-880f-40f1-8162-28ffed9574dd" (UID: "c0e8b085-880f-40f1-8162-28ffed9574dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:49:29.631123 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.631085    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0e8b085-880f-40f1-8162-28ffed9574dd" (UID: "c0e8b085-880f-40f1-8162-28ffed9574dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 15:49:29.631123 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.631113    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0e8b085-880f-40f1-8162-28ffed9574dd" (UID: "c0e8b085-880f-40f1-8162-28ffed9574dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:49:29.730163 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.730117    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-podinfo\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:29.730163 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.730155    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-token\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:29.730163 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.730165    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhpxc\" (UniqueName: \"kubernetes.io/projected/c0e8b085-880f-40f1-8162-28ffed9574dd-kube-api-access-jhpxc\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:29.730163 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:29.730174    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e8b085-880f-40f1-8162-28ffed9574dd-istio-envoy\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:49:30.254381 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:30.254339    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:30.562185 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:30.562110    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"
Sep 18 15:49:30.599146 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:30.599106    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"]
Sep 18 15:49:30.605459 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:30.605435    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-45956207-gw-openshift-default-b57d5695b-rjz79"]
Sep 18 15:49:31.090530 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:31.090496    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:31.743593 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:31.743562    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0e8b085-880f-40f1-8162-28ffed9574dd" path="/var/lib/kubelet/pods/c0e8b085-880f-40f1-8162-28ffed9574dd/volumes"
Sep 18 15:49:31.806612 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:31.806575    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:31.909100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:31.909047    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:32.329295 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.329242    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:32.393237 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.393203    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:49:32.423941 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.423911    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:32.459169 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.456018    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"]
Sep 18 15:49:32.459169 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.457008    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerName="istio-proxy"
Sep 18 15:49:32.459169 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.457028    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerName="istio-proxy"
Sep 18 15:49:32.459169 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.457065    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerName="istio-proxy"
Sep 18 15:49:32.459169 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.457073    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerName="istio-proxy"
Sep 18 15:49:32.459169 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.457251    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerName="istio-proxy"
Sep 18 15:49:32.464907 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.464876    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"
Sep 18 15:49:32.467837 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.467815    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d9680198\""
Sep 18 15:49:32.475623 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.475595    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"]
Sep 18 15:49:32.516375 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.516334    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"]
Sep 18 15:49:32.516879 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.516844    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e8b085-880f-40f1-8162-28ffed9574dd" containerName="istio-proxy"
Sep 18 15:49:32.519992 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.519969    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"
Sep 18 15:49:32.524318 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.524296    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d9680198-dockercfg-4rk5p\""
Sep 18 15:49:32.541564 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.541533    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"]
Sep 18 15:49:32.555640 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.555603    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xldqg\" (UniqueName: \"kubernetes.io/projected/b9c8db33-126f-4047-bb15-c2543035ce25-kube-api-access-xldqg\") pod \"payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2\" (UID: \"b9c8db33-126f-4047-bb15-c2543035ce25\") " pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"
Sep 18 15:49:32.555821 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.555766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c8db33-126f-4047-bb15-c2543035ce25-plugins-config-volume\") pod \"payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2\" (UID: \"b9c8db33-126f-4047-bb15-c2543035ce25\") " pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"
Sep 18 15:49:32.657379 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.657284    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c8db33-126f-4047-bb15-c2543035ce25-plugins-config-volume\") pod \"payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2\" (UID: \"b9c8db33-126f-4047-bb15-c2543035ce25\") " pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"
Sep 18 15:49:32.657379 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.657330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xldqg\" (UniqueName: \"kubernetes.io/projected/b9c8db33-126f-4047-bb15-c2543035ce25-kube-api-access-xldqg\") pod \"payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2\" (UID: \"b9c8db33-126f-4047-bb15-c2543035ce25\") " pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"
Sep 18 15:49:32.657379 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.657357    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73cd5e-199d-438b-9a56-6d63f53b6432-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2\" (UID: \"ed73cd5e-199d-438b-9a56-6d63f53b6432\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"
Sep 18 15:49:32.657690 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.657398    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlrws\" (UniqueName: \"kubernetes.io/projected/ed73cd5e-199d-438b-9a56-6d63f53b6432-kube-api-access-zlrws\") pod \"payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2\" (UID: \"ed73cd5e-199d-438b-9a56-6d63f53b6432\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"
Sep 18 15:49:32.658123 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.658101    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c8db33-126f-4047-bb15-c2543035ce25-plugins-config-volume\") pod \"payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2\" (UID: \"b9c8db33-126f-4047-bb15-c2543035ce25\") " pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"
Sep 18 15:49:32.666745 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.666716    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xldqg\" (UniqueName: \"kubernetes.io/projected/b9c8db33-126f-4047-bb15-c2543035ce25-kube-api-access-xldqg\") pod \"payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2\" (UID: \"b9c8db33-126f-4047-bb15-c2543035ce25\") " pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"
Sep 18 15:49:32.758295 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.758261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlrws\" (UniqueName: \"kubernetes.io/projected/ed73cd5e-199d-438b-9a56-6d63f53b6432-kube-api-access-zlrws\") pod \"payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2\" (UID: \"ed73cd5e-199d-438b-9a56-6d63f53b6432\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"
Sep 18 15:49:32.758478 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.758361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73cd5e-199d-438b-9a56-6d63f53b6432-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2\" (UID: \"ed73cd5e-199d-438b-9a56-6d63f53b6432\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"
Sep 18 15:49:32.759097 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.759074    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73cd5e-199d-438b-9a56-6d63f53b6432-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2\" (UID: \"ed73cd5e-199d-438b-9a56-6d63f53b6432\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"
Sep 18 15:49:32.768939 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.768910    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlrws\" (UniqueName: \"kubernetes.io/projected/ed73cd5e-199d-438b-9a56-6d63f53b6432-kube-api-access-zlrws\") pod \"payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2\" (UID: \"ed73cd5e-199d-438b-9a56-6d63f53b6432\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"
Sep 18 15:49:32.776621 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.776600    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"
Sep 18 15:49:32.831145 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:32.830792    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"
Sep 18 15:49:33.063319 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:33.063281    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"]
Sep 18 15:49:33.065497 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:49:33.065473    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c8db33_126f_4047_bb15_c2543035ce25.slice/crio-0e9bf584a1ffcdb5d2e69741803f5d6b370125fa37beb1a31fcc74552d05fb41 WatchSource:0}: Error finding container 0e9bf584a1ffcdb5d2e69741803f5d6b370125fa37beb1a31fcc74552d05fb41: Status 404 returned error can't find the container with id 0e9bf584a1ffcdb5d2e69741803f5d6b370125fa37beb1a31fcc74552d05fb41
Sep 18 15:49:33.137166 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:49:33.137138    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded73cd5e_199d_438b_9a56_6d63f53b6432.slice/crio-9e6642af54b859dd7f080830a8b23992dcd323f121338b62e7fee88b9e14c6fc WatchSource:0}: Error finding container 9e6642af54b859dd7f080830a8b23992dcd323f121338b62e7fee88b9e14c6fc: Status 404 returned error can't find the container with id 9e6642af54b859dd7f080830a8b23992dcd323f121338b62e7fee88b9e14c6fc
Sep 18 15:49:33.137469 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:33.137448    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"]
Sep 18 15:49:33.580340 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:33.580301    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2" event={"ID":"ed73cd5e-199d-438b-9a56-6d63f53b6432","Type":"ContainerStarted","Data":"cdd1466f5261bf2466c6c69ebdbd468770796e86b470e6f24c8254e8102f0365"}
Sep 18 15:49:33.580814 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:33.580376    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"
Sep 18 15:49:33.580814 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:33.580403    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2" event={"ID":"ed73cd5e-199d-438b-9a56-6d63f53b6432","Type":"ContainerStarted","Data":"9e6642af54b859dd7f080830a8b23992dcd323f121338b62e7fee88b9e14c6fc"}
Sep 18 15:49:33.581976 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:33.581953    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2" event={"ID":"b9c8db33-126f-4047-bb15-c2543035ce25","Type":"ContainerStarted","Data":"f553a8bad5ae4e5fb75bd525524ead8437579fb523d5c0283f2c506b4a347f43"}
Sep 18 15:49:33.582098 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:33.581991    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2" event={"ID":"b9c8db33-126f-4047-bb15-c2543035ce25","Type":"ContainerStarted","Data":"0e9bf584a1ffcdb5d2e69741803f5d6b370125fa37beb1a31fcc74552d05fb41"}
Sep 18 15:49:33.582098 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:33.582018    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"
Sep 18 15:49:33.603975 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:33.603920    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2" podStartSLOduration=1.603903791 podStartE2EDuration="1.603903791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:49:33.602576145 +0000 UTC m=+896.163808286" watchObservedRunningTime="2026-09-18 15:49:33.603903791 +0000 UTC m=+896.165135929"
Sep 18 15:49:33.624221 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:33.624163    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2" podStartSLOduration=1.624146026 podStartE2EDuration="1.624146026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:49:33.623468884 +0000 UTC m=+896.184701024" watchObservedRunningTime="2026-09-18 15:49:33.624146026 +0000 UTC m=+896.185378165"
Sep 18 15:49:44.590654 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:44.590619    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"
Sep 18 15:49:44.591115 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:44.590700    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"
Sep 18 15:49:45.185111 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:45.185075    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:46.877822 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:46.877788    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:49:49.667223 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:49.667125    2559 generic.go:356] "Generic (PLEG): container finished" podID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerID="d8720799028386081609a02e9b409643a6564f8fe3380ce2630e4f10053ae54a" exitCode=0
Sep 18 15:49:49.667223 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:49.667199    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" event={"ID":"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4","Type":"ContainerDied","Data":"d8720799028386081609a02e9b409643a6564f8fe3380ce2630e4f10053ae54a"}
Sep 18 15:49:49.667782 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:49.667730    2559 scope.go:117] "RemoveContainer" containerID="d8720799028386081609a02e9b409643a6564f8fe3380ce2630e4f10053ae54a"
Sep 18 15:49:49.668827 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:49.668795    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:49:49.668927 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:49.668879    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:49:49.668927 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:49.668912    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:49:50.678979 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:50.678935    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" event={"ID":"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4","Type":"ContainerStarted","Data":"19b21860feccff5508adafbdfba0149b96fa95df5e6399d712e12af288fb96cd"}
Sep 18 15:49:51.376969 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:51.376924    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:51.377479 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:51.377448    2559 patch_prober.go:28] interesting pod/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 18 15:49:51.377712 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:51.377671    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 18 15:49:52.377796 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:52.377748    2559 patch_prober.go:28] interesting pod/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 18 15:49:52.378369 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:52.377820    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 18 15:49:53.378071 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:53.378010    2559 patch_prober.go:28] interesting pod/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 18 15:49:53.378594 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:53.378103    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 18 15:49:53.990107 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:53.990069    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:54.377901 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:54.377766    2559 patch_prober.go:28] interesting pod/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 18 15:49:54.378237 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:54.378201    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 18 15:49:55.316642 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:55.316604    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:55.378204 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:55.377700    2559 patch_prober.go:28] interesting pod/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 18 15:49:55.378204 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:55.377767    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 18 15:49:56.035867 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:56.035828    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:56.381041 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:56.380965    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:56.381439 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:56.381234    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:56.382153 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:56.382130    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:49:56.895792 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:56.895762    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:57.595485 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:57.595451    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:58.173794 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:58.173743    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:58.740086 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:58.740053    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5"]
Sep 18 15:49:58.742727 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:58.742709    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5"
Sep 18 15:49:58.745513 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:58.745490    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 15:49:58.745611 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:58.745549    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 15:49:58.752441 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:58.752412    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5"]
Sep 18 15:49:58.820170 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:58.820126    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/288d5d37-3103-4daf-9c6d-3ee6cc670930-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d9680198-r88t5\" (UID: \"288d5d37-3103-4daf-9c6d-3ee6cc670930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5"
Sep 18 15:49:58.920810 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:58.920777    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/288d5d37-3103-4daf-9c6d-3ee6cc670930-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d9680198-r88t5\" (UID: \"288d5d37-3103-4daf-9c6d-3ee6cc670930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5"
Sep 18 15:49:58.921430 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:58.921409    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/288d5d37-3103-4daf-9c6d-3ee6cc670930-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d9680198-r88t5\" (UID: \"288d5d37-3103-4daf-9c6d-3ee6cc670930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5"
Sep 18 15:49:59.054432 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:59.054345    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5"
Sep 18 15:49:59.323033 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:59.323001    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5"]
Sep 18 15:49:59.324445 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:49:59.324422    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288d5d37_3103_4daf_9c6d_3ee6cc670930.slice/crio-06b1aa5c26b2897565724631f76975825bad1919eced8cccc53f3e718454c818 WatchSource:0}: Error finding container 06b1aa5c26b2897565724631f76975825bad1919eced8cccc53f3e718454c818: Status 404 returned error can't find the container with id 06b1aa5c26b2897565724631f76975825bad1919eced8cccc53f3e718454c818
Sep 18 15:49:59.735704 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:59.735666    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5" event={"ID":"288d5d37-3103-4daf-9c6d-3ee6cc670930","Type":"ContainerStarted","Data":"d96051573163b01d4f993af806738cc92a98b8cdbf5d4ff2e12f239acb8456f8"}
Sep 18 15:49:59.735997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:59.735757    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5" event={"ID":"288d5d37-3103-4daf-9c6d-3ee6cc670930","Type":"ContainerStarted","Data":"06b1aa5c26b2897565724631f76975825bad1919eced8cccc53f3e718454c818"}
Sep 18 15:49:59.746026 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:59.745982    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:49:59.756351 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:49:59.756301    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5" podStartSLOduration=1.756285946 podStartE2EDuration="1.756285946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:49:59.754206149 +0000 UTC m=+922.315438292" watchObservedRunningTime="2026-09-18 15:49:59.756285946 +0000 UTC m=+922.317518084"
Sep 18 15:50:00.575492 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:00.575451    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:00.742588 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:00.742552    2559 generic.go:356] "Generic (PLEG): container finished" podID="288d5d37-3103-4daf-9c6d-3ee6cc670930" containerID="d96051573163b01d4f993af806738cc92a98b8cdbf5d4ff2e12f239acb8456f8" exitCode=0
Sep 18 15:50:00.742766 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:00.742632    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5" event={"ID":"288d5d37-3103-4daf-9c6d-3ee6cc670930","Type":"ContainerDied","Data":"d96051573163b01d4f993af806738cc92a98b8cdbf5d4ff2e12f239acb8456f8"}
Sep 18 15:50:01.530565 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:01.530528    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:01.575163 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:01.575130    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:50:01.889899 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:01.889874    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5"
Sep 18 15:50:01.953124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:01.953081    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/288d5d37-3103-4daf-9c6d-3ee6cc670930-maas-api-service-ca\") pod \"288d5d37-3103-4daf-9c6d-3ee6cc670930\" (UID: \"288d5d37-3103-4daf-9c6d-3ee6cc670930\") "
Sep 18 15:50:01.953442 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:01.953414    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288d5d37-3103-4daf-9c6d-3ee6cc670930-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "288d5d37-3103-4daf-9c6d-3ee6cc670930" (UID: "288d5d37-3103-4daf-9c6d-3ee6cc670930"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:50:02.054625 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:02.054531    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/288d5d37-3103-4daf-9c6d-3ee6cc670930-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:02.237360 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:02.237326    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:02.754550 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:02.754516    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5"
Sep 18 15:50:02.754550 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:02.754539    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5" event={"ID":"288d5d37-3103-4daf-9c6d-3ee6cc670930","Type":"ContainerDied","Data":"06b1aa5c26b2897565724631f76975825bad1919eced8cccc53f3e718454c818"}
Sep 18 15:50:02.755011 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:02.754586    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06b1aa5c26b2897565724631f76975825bad1919eced8cccc53f3e718454c818"
Sep 18 15:50:02.800213 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:02.800171    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:03.683488 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:03.683454    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:04.420450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.420406    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"]
Sep 18 15:50:04.421052 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.421032    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="288d5d37-3103-4daf-9c6d-3ee6cc670930" containerName="revoke-keys"
Sep 18 15:50:04.421150 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.421054    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="288d5d37-3103-4daf-9c6d-3ee6cc670930" containerName="revoke-keys"
Sep 18 15:50:04.421206 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.421152    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="288d5d37-3103-4daf-9c6d-3ee6cc670930" containerName="revoke-keys"
Sep 18 15:50:04.424312 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.424289    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.428515 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.428493    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-998130\"/\"default-dockercfg-lv7jt\""
Sep 18 15:50:04.428638 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.428493    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-998130\"/\"kube-root-ca.crt\""
Sep 18 15:50:04.428638 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.428494    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-998130\"/\"openshift-service-ca.crt\""
Sep 18 15:50:04.428638 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.428542    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-998130\"/\"premium-w0-998130-kserve-self-signed-certs\""
Sep 18 15:50:04.436042 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.436019    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"]
Sep 18 15:50:04.454139 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.454106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:04.477954 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.477919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k42jk\" (UniqueName: \"kubernetes.io/projected/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-kube-api-access-k42jk\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.477954 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.477954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-tmp-dir\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.478148 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.477978    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-home\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.478148 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.478092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-dshm\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.478148 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.478115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-tls-certs\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.478254 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.478206    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-model-cache\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.579152 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.579118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-model-cache\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.579334 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.579185    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k42jk\" (UniqueName: \"kubernetes.io/projected/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-kube-api-access-k42jk\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.579334 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.579217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-tmp-dir\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.579334 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.579247    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-home\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.579334 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.579307    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-dshm\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.579334 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.579332    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-tls-certs\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.579613 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.579593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-model-cache\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.579674 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.579638    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-tmp-dir\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.579728 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.579680    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-home\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.581609 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.581582    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-dshm\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.581762 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.581746    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-tls-certs\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.588599 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.588575    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k42jk\" (UniqueName: \"kubernetes.io/projected/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-kube-api-access-k42jk\") pod \"premium-w0-998130-kserve-6cbc66fc94-hfjhr\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") " pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:04.733462 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:04.733424    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:05.013662 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:05.013636    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"]
Sep 18 15:50:05.015788 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:50:05.015758    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda989b04c_6b87_4ae3_a66d_b3841bfcbf5b.slice/crio-1ee4b08870605e6d7bc228d3f895c0afbf36f35d5825981310b9448771fcee7d WatchSource:0}: Error finding container 1ee4b08870605e6d7bc228d3f895c0afbf36f35d5825981310b9448771fcee7d: Status 404 returned error can't find the container with id 1ee4b08870605e6d7bc228d3f895c0afbf36f35d5825981310b9448771fcee7d
Sep 18 15:50:05.773513 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:05.773475    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr" event={"ID":"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b","Type":"ContainerStarted","Data":"d6df472a06dae3cef33f6600e40ec79b2120d4772fb15d675c1078df8bad0bb7"}
Sep 18 15:50:05.773513 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:05.773518    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr" event={"ID":"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b","Type":"ContainerStarted","Data":"1ee4b08870605e6d7bc228d3f895c0afbf36f35d5825981310b9448771fcee7d"}
Sep 18 15:50:05.797923 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:05.797874    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr" podStartSLOduration=1.797844947 podStartE2EDuration="1.797844947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:50:05.795393231 +0000 UTC m=+928.356625387" watchObservedRunningTime="2026-09-18 15:50:05.797844947 +0000 UTC m=+928.359077085"
Sep 18 15:50:09.841544 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:09.841505    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:09.863304 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:09.863258    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"]
Sep 18 15:50:09.863599 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:09.863568    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2" podUID="b9c8db33-126f-4047-bb15-c2543035ce25" containerName="payload-processing" containerID="cri-o://f553a8bad5ae4e5fb75bd525524ead8437579fb523d5c0283f2c506b4a347f43" gracePeriod=30
Sep 18 15:50:09.869506 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:09.869478    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"]
Sep 18 15:50:09.869693 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:09.869663    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2" podUID="ed73cd5e-199d-438b-9a56-6d63f53b6432" containerName="payload-pre-processing" containerID="cri-o://cdd1466f5261bf2466c6c69ebdbd468770796e86b470e6f24c8254e8102f0365" gracePeriod=30
Sep 18 15:50:09.911889 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:09.911819    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:09.981453 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:09.981420    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:10.067921 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.067885    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5"]
Sep 18 15:50:10.074791 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.074746    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9680198-r88t5"]
Sep 18 15:50:10.075609 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.075577    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:10.115940 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.115910    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"
Sep 18 15:50:10.147995 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.147962    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:10.151501 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.151483    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"
Sep 18 15:50:10.233988 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.233957    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73cd5e-199d-438b-9a56-6d63f53b6432-plugins-config-volume\") pod \"ed73cd5e-199d-438b-9a56-6d63f53b6432\" (UID: \"ed73cd5e-199d-438b-9a56-6d63f53b6432\") "
Sep 18 15:50:10.234190 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.234077    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlrws\" (UniqueName: \"kubernetes.io/projected/ed73cd5e-199d-438b-9a56-6d63f53b6432-kube-api-access-zlrws\") pod \"ed73cd5e-199d-438b-9a56-6d63f53b6432\" (UID: \"ed73cd5e-199d-438b-9a56-6d63f53b6432\") "
Sep 18 15:50:10.234190 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.234137    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xldqg\" (UniqueName: \"kubernetes.io/projected/b9c8db33-126f-4047-bb15-c2543035ce25-kube-api-access-xldqg\") pod \"b9c8db33-126f-4047-bb15-c2543035ce25\" (UID: \"b9c8db33-126f-4047-bb15-c2543035ce25\") "
Sep 18 15:50:10.234190 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.234168    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c8db33-126f-4047-bb15-c2543035ce25-plugins-config-volume\") pod \"b9c8db33-126f-4047-bb15-c2543035ce25\" (UID: \"b9c8db33-126f-4047-bb15-c2543035ce25\") "
Sep 18 15:50:10.234383 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.234360    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed73cd5e-199d-438b-9a56-6d63f53b6432-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed73cd5e-199d-438b-9a56-6d63f53b6432" (UID: "ed73cd5e-199d-438b-9a56-6d63f53b6432"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:50:10.234531 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.234445    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73cd5e-199d-438b-9a56-6d63f53b6432-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:10.234643 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.234613    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c8db33-126f-4047-bb15-c2543035ce25-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9c8db33-126f-4047-bb15-c2543035ce25" (UID: "b9c8db33-126f-4047-bb15-c2543035ce25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:50:10.236171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.236152    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed73cd5e-199d-438b-9a56-6d63f53b6432-kube-api-access-zlrws" (OuterVolumeSpecName: "kube-api-access-zlrws") pod "ed73cd5e-199d-438b-9a56-6d63f53b6432" (UID: "ed73cd5e-199d-438b-9a56-6d63f53b6432"). InnerVolumeSpecName "kube-api-access-zlrws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:50:10.236253 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.236225    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c8db33-126f-4047-bb15-c2543035ce25-kube-api-access-xldqg" (OuterVolumeSpecName: "kube-api-access-xldqg") pod "b9c8db33-126f-4047-bb15-c2543035ce25" (UID: "b9c8db33-126f-4047-bb15-c2543035ce25"). InnerVolumeSpecName "kube-api-access-xldqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:50:10.335894 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.335835    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlrws\" (UniqueName: \"kubernetes.io/projected/ed73cd5e-199d-438b-9a56-6d63f53b6432-kube-api-access-zlrws\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:10.335894 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.335888    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xldqg\" (UniqueName: \"kubernetes.io/projected/b9c8db33-126f-4047-bb15-c2543035ce25-kube-api-access-xldqg\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:10.335894 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.335897    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c8db33-126f-4047-bb15-c2543035ce25-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:10.499757 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.499718    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:10.801813 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.801721    2559 generic.go:356] "Generic (PLEG): container finished" podID="ed73cd5e-199d-438b-9a56-6d63f53b6432" containerID="cdd1466f5261bf2466c6c69ebdbd468770796e86b470e6f24c8254e8102f0365" exitCode=0
Sep 18 15:50:10.801813 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.801767    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"
Sep 18 15:50:10.802089 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.801809    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2" event={"ID":"ed73cd5e-199d-438b-9a56-6d63f53b6432","Type":"ContainerDied","Data":"cdd1466f5261bf2466c6c69ebdbd468770796e86b470e6f24c8254e8102f0365"}
Sep 18 15:50:10.802089 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.801888    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2" event={"ID":"ed73cd5e-199d-438b-9a56-6d63f53b6432","Type":"ContainerDied","Data":"9e6642af54b859dd7f080830a8b23992dcd323f121338b62e7fee88b9e14c6fc"}
Sep 18 15:50:10.802089 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.801921    2559 scope.go:117] "RemoveContainer" containerID="cdd1466f5261bf2466c6c69ebdbd468770796e86b470e6f24c8254e8102f0365"
Sep 18 15:50:10.803291 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.803267    2559 generic.go:356] "Generic (PLEG): container finished" podID="b9c8db33-126f-4047-bb15-c2543035ce25" containerID="f553a8bad5ae4e5fb75bd525524ead8437579fb523d5c0283f2c506b4a347f43" exitCode=0
Sep 18 15:50:10.803394 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.803329    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"
Sep 18 15:50:10.803394 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.803361    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2" event={"ID":"b9c8db33-126f-4047-bb15-c2543035ce25","Type":"ContainerDied","Data":"f553a8bad5ae4e5fb75bd525524ead8437579fb523d5c0283f2c506b4a347f43"}
Sep 18 15:50:10.803463 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.803429    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2" event={"ID":"b9c8db33-126f-4047-bb15-c2543035ce25","Type":"ContainerDied","Data":"0e9bf584a1ffcdb5d2e69741803f5d6b370125fa37beb1a31fcc74552d05fb41"}
Sep 18 15:50:10.811542 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.811519    2559 scope.go:117] "RemoveContainer" containerID="cdd1466f5261bf2466c6c69ebdbd468770796e86b470e6f24c8254e8102f0365"
Sep 18 15:50:10.811922 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:50:10.811895    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd1466f5261bf2466c6c69ebdbd468770796e86b470e6f24c8254e8102f0365\": container with ID starting with cdd1466f5261bf2466c6c69ebdbd468770796e86b470e6f24c8254e8102f0365 not found: ID does not exist" containerID="cdd1466f5261bf2466c6c69ebdbd468770796e86b470e6f24c8254e8102f0365"
Sep 18 15:50:10.812034 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.811931    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd1466f5261bf2466c6c69ebdbd468770796e86b470e6f24c8254e8102f0365"} err="failed to get container status \"cdd1466f5261bf2466c6c69ebdbd468770796e86b470e6f24c8254e8102f0365\": rpc error: code = NotFound desc = could not find container \"cdd1466f5261bf2466c6c69ebdbd468770796e86b470e6f24c8254e8102f0365\": container with ID starting with cdd1466f5261bf2466c6c69ebdbd468770796e86b470e6f24c8254e8102f0365 not found: ID does not exist"
Sep 18 15:50:10.812034 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.811959    2559 scope.go:117] "RemoveContainer" containerID="f553a8bad5ae4e5fb75bd525524ead8437579fb523d5c0283f2c506b4a347f43"
Sep 18 15:50:10.821361 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.821343    2559 scope.go:117] "RemoveContainer" containerID="f553a8bad5ae4e5fb75bd525524ead8437579fb523d5c0283f2c506b4a347f43"
Sep 18 15:50:10.821672 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:50:10.821654    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f553a8bad5ae4e5fb75bd525524ead8437579fb523d5c0283f2c506b4a347f43\": container with ID starting with f553a8bad5ae4e5fb75bd525524ead8437579fb523d5c0283f2c506b4a347f43 not found: ID does not exist" containerID="f553a8bad5ae4e5fb75bd525524ead8437579fb523d5c0283f2c506b4a347f43"
Sep 18 15:50:10.821714 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.821679    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f553a8bad5ae4e5fb75bd525524ead8437579fb523d5c0283f2c506b4a347f43"} err="failed to get container status \"f553a8bad5ae4e5fb75bd525524ead8437579fb523d5c0283f2c506b4a347f43\": rpc error: code = NotFound desc = could not find container \"f553a8bad5ae4e5fb75bd525524ead8437579fb523d5c0283f2c506b4a347f43\": container with ID starting with f553a8bad5ae4e5fb75bd525524ead8437579fb523d5c0283f2c506b4a347f43 not found: ID does not exist"
Sep 18 15:50:10.830174 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.830139    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"]
Sep 18 15:50:10.835229 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.835202    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d9680198-7b8d545f97-8xlm2"]
Sep 18 15:50:10.846577 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.846543    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"]
Sep 18 15:50:10.853813 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:10.853786    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d9680198-7dfbbfbcdb-jx6b2"]
Sep 18 15:50:11.094374 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:11.094279    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:11.172202 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:11.172053    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"]
Sep 18 15:50:11.172408 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:11.172338    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerName="istio-proxy" containerID="cri-o://19b21860feccff5508adafbdfba0149b96fa95df5e6399d712e12af288fb96cd" gracePeriod=30
Sep 18 15:50:11.196606 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:11.196569    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:11.384124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:11.384036    2559 patch_prober.go:28] interesting pod/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 15:50:11.384124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:11.384097    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 15:50:11.595609 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:11.595577    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:11.744459 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:11.744420    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="288d5d37-3103-4daf-9c6d-3ee6cc670930" path="/var/lib/kubelet/pods/288d5d37-3103-4daf-9c6d-3ee6cc670930/volumes"
Sep 18 15:50:11.744804 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:11.744791    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9c8db33-126f-4047-bb15-c2543035ce25" path="/var/lib/kubelet/pods/b9c8db33-126f-4047-bb15-c2543035ce25/volumes"
Sep 18 15:50:11.745170 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:11.745158    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed73cd5e-199d-438b-9a56-6d63f53b6432" path="/var/lib/kubelet/pods/ed73cd5e-199d-438b-9a56-6d63f53b6432/volumes"
Sep 18 15:50:11.905456 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:11.905410    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:12.087327 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:12.087230    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:12.232593 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:12.232556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:12.924432 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:12.924384    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:13.784755 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:13.784721    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:14.734488 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:14.734447    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:14.734933 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:14.734512    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:14.747232 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:14.747204    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:14.839391 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:14.839363    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:50:15.118609 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:15.118511    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:15.555754 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:15.555712    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:16.426000 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.425976    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:50:16.510758 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.510673    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:16.594960 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.594926    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4vqv\" (UniqueName: \"kubernetes.io/projected/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-kube-api-access-q4vqv\") pod \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") "
Sep 18 15:50:16.595124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.594977    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istiod-ca-cert\") pod \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") "
Sep 18 15:50:16.595124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595030    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-workload-certs\") pod \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") "
Sep 18 15:50:16.595124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595052    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-credential-socket\") pod \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") "
Sep 18 15:50:16.595124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595088    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-envoy\") pod \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") "
Sep 18 15:50:16.595124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595109    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-token\") pod \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") "
Sep 18 15:50:16.595397 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595127    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-data\") pod \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") "
Sep 18 15:50:16.595397 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595374    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" (UID: "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:50:16.595511 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595393    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" (UID: "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:50:16.595511 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595443    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" (UID: "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:50:16.595511 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595461    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" (UID: "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:50:16.595511 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595467    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-podinfo\") pod \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") "
Sep 18 15:50:16.595712 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595529    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-workload-socket\") pod \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\" (UID: \"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4\") "
Sep 18 15:50:16.595771 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595722    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" (UID: "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:50:16.595925 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595906    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-data\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:16.595994 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595931    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-workload-socket\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:16.595994 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595944    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istiod-ca-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:16.595994 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595957    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-workload-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:16.595994 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.595970    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-credential-socket\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:16.597371 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.597345    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" (UID: "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:50:16.597458 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.597398    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" (UID: "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:50:16.597602 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.597581    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-kube-api-access-q4vqv" (OuterVolumeSpecName: "kube-api-access-q4vqv") pod "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" (UID: "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4"). InnerVolumeSpecName "kube-api-access-q4vqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:50:16.597602 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.597595    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" (UID: "67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 15:50:16.696928 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.696889    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-podinfo\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:16.696928 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.696923    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4vqv\" (UniqueName: \"kubernetes.io/projected/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-kube-api-access-q4vqv\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:16.696928 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.696932    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-envoy\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:16.697164 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.696940    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4-istio-token\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:16.839230 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.839144    2559 generic.go:356] "Generic (PLEG): container finished" podID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerID="19b21860feccff5508adafbdfba0149b96fa95df5e6399d712e12af288fb96cd" exitCode=0
Sep 18 15:50:16.839230 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.839198    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"
Sep 18 15:50:16.839420 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.839236    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" event={"ID":"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4","Type":"ContainerDied","Data":"19b21860feccff5508adafbdfba0149b96fa95df5e6399d712e12af288fb96cd"}
Sep 18 15:50:16.839420 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.839310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m" event={"ID":"67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4","Type":"ContainerDied","Data":"b6012be1c71ad709f64f6821a6755f79c15078f1c0b1ddb2104d694ac47b86a3"}
Sep 18 15:50:16.839420 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.839338    2559 scope.go:117] "RemoveContainer" containerID="19b21860feccff5508adafbdfba0149b96fa95df5e6399d712e12af288fb96cd"
Sep 18 15:50:16.848505 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.848488    2559 scope.go:117] "RemoveContainer" containerID="d8720799028386081609a02e9b409643a6564f8fe3380ce2630e4f10053ae54a"
Sep 18 15:50:16.856347 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.856328    2559 scope.go:117] "RemoveContainer" containerID="19b21860feccff5508adafbdfba0149b96fa95df5e6399d712e12af288fb96cd"
Sep 18 15:50:16.856594 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:50:16.856575    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b21860feccff5508adafbdfba0149b96fa95df5e6399d712e12af288fb96cd\": container with ID starting with 19b21860feccff5508adafbdfba0149b96fa95df5e6399d712e12af288fb96cd not found: ID does not exist" containerID="19b21860feccff5508adafbdfba0149b96fa95df5e6399d712e12af288fb96cd"
Sep 18 15:50:16.856631 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.856602    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b21860feccff5508adafbdfba0149b96fa95df5e6399d712e12af288fb96cd"} err="failed to get container status \"19b21860feccff5508adafbdfba0149b96fa95df5e6399d712e12af288fb96cd\": rpc error: code = NotFound desc = could not find container \"19b21860feccff5508adafbdfba0149b96fa95df5e6399d712e12af288fb96cd\": container with ID starting with 19b21860feccff5508adafbdfba0149b96fa95df5e6399d712e12af288fb96cd not found: ID does not exist"
Sep 18 15:50:16.856631 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.856618    2559 scope.go:117] "RemoveContainer" containerID="d8720799028386081609a02e9b409643a6564f8fe3380ce2630e4f10053ae54a"
Sep 18 15:50:16.856896 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:50:16.856877    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8720799028386081609a02e9b409643a6564f8fe3380ce2630e4f10053ae54a\": container with ID starting with d8720799028386081609a02e9b409643a6564f8fe3380ce2630e4f10053ae54a not found: ID does not exist" containerID="d8720799028386081609a02e9b409643a6564f8fe3380ce2630e4f10053ae54a"
Sep 18 15:50:16.856965 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.856900    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8720799028386081609a02e9b409643a6564f8fe3380ce2630e4f10053ae54a"} err="failed to get container status \"d8720799028386081609a02e9b409643a6564f8fe3380ce2630e4f10053ae54a\": rpc error: code = NotFound desc = could not find container \"d8720799028386081609a02e9b409643a6564f8fe3380ce2630e4f10053ae54a\": container with ID starting with d8720799028386081609a02e9b409643a6564f8fe3380ce2630e4f10053ae54a not found: ID does not exist"
Sep 18 15:50:16.875316 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.875272    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"]
Sep 18 15:50:16.881281 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:16.881252    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d9680198-openshift-default-579ff58d8b-lc24m"]
Sep 18 15:50:17.750652 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:17.744440    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" path="/var/lib/kubelet/pods/67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4/volumes"
Sep 18 15:50:17.791738 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:17.791695    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:18.604733 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:18.604695    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:19.332747 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.332715    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:19.430571 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.430520    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:19.761555 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.761514    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:19.861716 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.861683    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:19.917396 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.917367    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"]
Sep 18 15:50:19.917836 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.917816    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerName="istio-proxy"
Sep 18 15:50:19.917836 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.917835    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerName="istio-proxy"
Sep 18 15:50:19.917836 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.917850    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c8db33-126f-4047-bb15-c2543035ce25" containerName="payload-processing"
Sep 18 15:50:19.918059 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.917871    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c8db33-126f-4047-bb15-c2543035ce25" containerName="payload-processing"
Sep 18 15:50:19.918059 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.917884    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerName="istio-proxy"
Sep 18 15:50:19.918059 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.917892    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerName="istio-proxy"
Sep 18 15:50:19.918059 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.917901    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed73cd5e-199d-438b-9a56-6d63f53b6432" containerName="payload-pre-processing"
Sep 18 15:50:19.918059 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.917910    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed73cd5e-199d-438b-9a56-6d63f53b6432" containerName="payload-pre-processing"
Sep 18 15:50:19.918059 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.917991    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerName="istio-proxy"
Sep 18 15:50:19.918059 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.918002    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed73cd5e-199d-438b-9a56-6d63f53b6432" containerName="payload-pre-processing"
Sep 18 15:50:19.918059 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.918020    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e78b34-9bf9-4bd8-be20-c6a3fbf4a0f4" containerName="istio-proxy"
Sep 18 15:50:19.918059 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.918031    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c8db33-126f-4047-bb15-c2543035ce25" containerName="payload-processing"
Sep 18 15:50:19.922572 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.922549    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"
Sep 18 15:50:19.925148 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.925126    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c9ad8682\""
Sep 18 15:50:19.933831 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.933805    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"]
Sep 18 15:50:19.934959 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.934927    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:19.963917 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.963881    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"]
Sep 18 15:50:19.967461 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.967438    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"
Sep 18 15:50:19.970763 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.970744    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c9ad8682-dockercfg-lwp4x\""
Sep 18 15:50:19.985021 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:19.984986    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"]
Sep 18 15:50:20.026407 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.026308    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf5qk\" (UniqueName: \"kubernetes.io/projected/194618f7-406f-492c-9e5e-146fa5f6d171-kube-api-access-pf5qk\") pod \"payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx\" (UID: \"194618f7-406f-492c-9e5e-146fa5f6d171\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"
Sep 18 15:50:20.026407 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.026361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194618f7-406f-492c-9e5e-146fa5f6d171-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx\" (UID: \"194618f7-406f-492c-9e5e-146fa5f6d171\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"
Sep 18 15:50:20.127260 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.127222    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da41811-bbc6-4bf9-ac26-89b143b20029-plugins-config-volume\") pod \"payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs\" (UID: \"1da41811-bbc6-4bf9-ac26-89b143b20029\") " pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"
Sep 18 15:50:20.127427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.127273    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9slt\" (UniqueName: \"kubernetes.io/projected/1da41811-bbc6-4bf9-ac26-89b143b20029-kube-api-access-h9slt\") pod \"payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs\" (UID: \"1da41811-bbc6-4bf9-ac26-89b143b20029\") " pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"
Sep 18 15:50:20.127427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.127337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf5qk\" (UniqueName: \"kubernetes.io/projected/194618f7-406f-492c-9e5e-146fa5f6d171-kube-api-access-pf5qk\") pod \"payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx\" (UID: \"194618f7-406f-492c-9e5e-146fa5f6d171\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"
Sep 18 15:50:20.127427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.127372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194618f7-406f-492c-9e5e-146fa5f6d171-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx\" (UID: \"194618f7-406f-492c-9e5e-146fa5f6d171\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"
Sep 18 15:50:20.128107 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.128088    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194618f7-406f-492c-9e5e-146fa5f6d171-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx\" (UID: \"194618f7-406f-492c-9e5e-146fa5f6d171\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"
Sep 18 15:50:20.136486 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.136455    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf5qk\" (UniqueName: \"kubernetes.io/projected/194618f7-406f-492c-9e5e-146fa5f6d171-kube-api-access-pf5qk\") pod \"payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx\" (UID: \"194618f7-406f-492c-9e5e-146fa5f6d171\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"
Sep 18 15:50:20.227920 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.227848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da41811-bbc6-4bf9-ac26-89b143b20029-plugins-config-volume\") pod \"payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs\" (UID: \"1da41811-bbc6-4bf9-ac26-89b143b20029\") " pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"
Sep 18 15:50:20.228100 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.227942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9slt\" (UniqueName: \"kubernetes.io/projected/1da41811-bbc6-4bf9-ac26-89b143b20029-kube-api-access-h9slt\") pod \"payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs\" (UID: \"1da41811-bbc6-4bf9-ac26-89b143b20029\") " pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"
Sep 18 15:50:20.228552 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.228529    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da41811-bbc6-4bf9-ac26-89b143b20029-plugins-config-volume\") pod \"payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs\" (UID: \"1da41811-bbc6-4bf9-ac26-89b143b20029\") " pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"
Sep 18 15:50:20.233787 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.233758    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"
Sep 18 15:50:20.236162 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.236134    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9slt\" (UniqueName: \"kubernetes.io/projected/1da41811-bbc6-4bf9-ac26-89b143b20029-kube-api-access-h9slt\") pod \"payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs\" (UID: \"1da41811-bbc6-4bf9-ac26-89b143b20029\") " pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"
Sep 18 15:50:20.278398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.278307    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"
Sep 18 15:50:20.507682 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.507655    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"]
Sep 18 15:50:20.509575 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:50:20.509548    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194618f7_406f_492c_9e5e_146fa5f6d171.slice/crio-3b5b4fdc1ca68068f0726cd8e14df6309fea52ee29fd5dd6c9277a70830ce587 WatchSource:0}: Error finding container 3b5b4fdc1ca68068f0726cd8e14df6309fea52ee29fd5dd6c9277a70830ce587: Status 404 returned error can't find the container with id 3b5b4fdc1ca68068f0726cd8e14df6309fea52ee29fd5dd6c9277a70830ce587
Sep 18 15:50:20.562189 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.562101    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"]
Sep 18 15:50:20.563792 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:50:20.563759    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da41811_bbc6_4bf9_ac26_89b143b20029.slice/crio-62ab192eb3483b6d7b58a30df18fdddc1b33ad2cc782a37377c21661296e9520 WatchSource:0}: Error finding container 62ab192eb3483b6d7b58a30df18fdddc1b33ad2cc782a37377c21661296e9520: Status 404 returned error can't find the container with id 62ab192eb3483b6d7b58a30df18fdddc1b33ad2cc782a37377c21661296e9520
Sep 18 15:50:20.867574 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.867467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx" event={"ID":"194618f7-406f-492c-9e5e-146fa5f6d171","Type":"ContainerStarted","Data":"e9b430e041012c3d66ff5f6a35e267964c5265fbfe9da5e41af941c4a526328d"}
Sep 18 15:50:20.867574 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.867540    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"
Sep 18 15:50:20.867574 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.867565    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx" event={"ID":"194618f7-406f-492c-9e5e-146fa5f6d171","Type":"ContainerStarted","Data":"3b5b4fdc1ca68068f0726cd8e14df6309fea52ee29fd5dd6c9277a70830ce587"}
Sep 18 15:50:20.869249 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.869222    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs" event={"ID":"1da41811-bbc6-4bf9-ac26-89b143b20029","Type":"ContainerStarted","Data":"a3ef493f74ea7514ebcd331178e93fdff65e3b7940352cf30cd065bf67235b64"}
Sep 18 15:50:20.869393 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.869266    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs" event={"ID":"1da41811-bbc6-4bf9-ac26-89b143b20029","Type":"ContainerStarted","Data":"62ab192eb3483b6d7b58a30df18fdddc1b33ad2cc782a37377c21661296e9520"}
Sep 18 15:50:20.869393 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.869320    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"
Sep 18 15:50:20.886078 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.885996    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx" podStartSLOduration=1.8859801919999999 podStartE2EDuration="1.885980192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:50:20.883797854 +0000 UTC m=+943.445030019" watchObservedRunningTime="2026-09-18 15:50:20.885980192 +0000 UTC m=+943.447212331"
Sep 18 15:50:20.904618 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:20.904555    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs" podStartSLOduration=1.904533263 podStartE2EDuration="1.904533263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:50:20.901879115 +0000 UTC m=+943.463111250" watchObservedRunningTime="2026-09-18 15:50:20.904533263 +0000 UTC m=+943.465765409"
Sep 18 15:50:22.074353 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:22.074320    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:50:30.652017 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:30.651979    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:31.877010 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:31.876980    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"
Sep 18 15:50:31.877390 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:31.877061    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"
Sep 18 15:50:35.139833 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.139800    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"]
Sep 18 15:50:35.143532 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.143515    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.148258 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.148231    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-70c6df\"/\"kube-root-ca.crt\""
Sep 18 15:50:35.148384 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.148318    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-70c6df\"/\"test-model-70c6df-kserve-self-signed-certs\""
Sep 18 15:50:35.148384 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.148318    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-70c6df\"/\"default-dockercfg-zwd85\""
Sep 18 15:50:35.148514 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.148344    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-70c6df\"/\"openshift-service-ca.crt\""
Sep 18 15:50:35.155377 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.155348    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"]
Sep 18 15:50:35.169520 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.169485    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-tmp-dir\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.169695 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.169532    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8be15b0-a494-4ff4-a6e2-98048350a522-tls-certs\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.169738 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.169701    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-model-cache\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.169783 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.169753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rx6l\" (UniqueName: \"kubernetes.io/projected/c8be15b0-a494-4ff4-a6e2-98048350a522-kube-api-access-7rx6l\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.169832 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.169792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-home\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.169832 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.169818    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-dshm\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.182396 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.182353    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:35.270953 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.270913    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-model-cache\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.271170 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.270974    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rx6l\" (UniqueName: \"kubernetes.io/projected/c8be15b0-a494-4ff4-a6e2-98048350a522-kube-api-access-7rx6l\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.271170 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.271001    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-home\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.271170 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.271022    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-dshm\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.271170 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.271101    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-tmp-dir\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.271170 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.271125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8be15b0-a494-4ff4-a6e2-98048350a522-tls-certs\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.271456 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.271432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-home\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.271515 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.271456    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-model-cache\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.271574 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.271528    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-tmp-dir\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.273405 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.273381    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-dshm\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.273632 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.273614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8be15b0-a494-4ff4-a6e2-98048350a522-tls-certs\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.280486 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.280461    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rx6l\" (UniqueName: \"kubernetes.io/projected/c8be15b0-a494-4ff4-a6e2-98048350a522-kube-api-access-7rx6l\") pod \"test-model-70c6df-kserve-d6b67888f-h6rb4\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") " pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.454372 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.454334    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:35.728237 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.728202    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"]
Sep 18 15:50:35.729824 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:50:35.729797    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8be15b0_a494_4ff4_a6e2_98048350a522.slice/crio-2e853e0f81a2b58741843f9cccc4dc14b39c96a55b6234c512ec3a025fcefeeb WatchSource:0}: Error finding container 2e853e0f81a2b58741843f9cccc4dc14b39c96a55b6234c512ec3a025fcefeeb: Status 404 returned error can't find the container with id 2e853e0f81a2b58741843f9cccc4dc14b39c96a55b6234c512ec3a025fcefeeb
Sep 18 15:50:35.954587 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.954552    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4" event={"ID":"c8be15b0-a494-4ff4-a6e2-98048350a522","Type":"ContainerStarted","Data":"7ec4399364995cc5446926966be3803c134006aba72cbefe2260cf53895efaf2"}
Sep 18 15:50:35.954771 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.954598    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4" event={"ID":"c8be15b0-a494-4ff4-a6e2-98048350a522","Type":"ContainerStarted","Data":"2e853e0f81a2b58741843f9cccc4dc14b39c96a55b6234c512ec3a025fcefeeb"}
Sep 18 15:50:35.975975 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:35.975921    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4" podStartSLOduration=0.975902413 podStartE2EDuration="975.902413ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:50:35.974233038 +0000 UTC m=+958.535465179" watchObservedRunningTime="2026-09-18 15:50:35.975902413 +0000 UTC m=+958.537134552"
Sep 18 15:50:41.571867 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:41.571809    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:42.403688 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:42.403651    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:42.692320 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:42.692289    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:50:43.211527 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:43.211494    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:43.964776 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:43.964738    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:44.611169 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:44.611133    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:44.646501 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:44.646462    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s"]
Sep 18 15:50:44.651789 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:44.651766    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s"
Sep 18 15:50:44.654533 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:44.654508    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 15:50:44.654693 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:44.654674    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 15:50:44.661728 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:44.661702    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s"]
Sep 18 15:50:44.760082 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:44.760047    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd72f4bb-b404-433d-a80b-2ce7e5bed685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s\" (UID: \"cd72f4bb-b404-433d-a80b-2ce7e5bed685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s"
Sep 18 15:50:44.861262 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:44.861169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd72f4bb-b404-433d-a80b-2ce7e5bed685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s\" (UID: \"cd72f4bb-b404-433d-a80b-2ce7e5bed685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s"
Sep 18 15:50:44.861773 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:44.861753    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd72f4bb-b404-433d-a80b-2ce7e5bed685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s\" (UID: \"cd72f4bb-b404-433d-a80b-2ce7e5bed685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s"
Sep 18 15:50:44.964464 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:44.964435    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s"
Sep 18 15:50:45.236928 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:45.236904    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s"]
Sep 18 15:50:45.239261 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:50:45.239232    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd72f4bb_b404_433d_a80b_2ce7e5bed685.slice/crio-d00ee6bcda1c19fa6d3960e75aca7dd9d33497cfcd27eddfbc6aa605bb837b1e WatchSource:0}: Error finding container d00ee6bcda1c19fa6d3960e75aca7dd9d33497cfcd27eddfbc6aa605bb837b1e: Status 404 returned error can't find the container with id d00ee6bcda1c19fa6d3960e75aca7dd9d33497cfcd27eddfbc6aa605bb837b1e
Sep 18 15:50:45.454427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:45.454391    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:45.454624 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:45.454468    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:45.468010 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:45.467987    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:45.686658 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:45.686617    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:46.008225 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:46.008192    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd72f4bb-b404-433d-a80b-2ce7e5bed685" containerID="f5167823de44819f829ccc24c060118427debd3d3d425fd4d801ab49eee6c570" exitCode=0
Sep 18 15:50:46.008426 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:46.008286    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s" event={"ID":"cd72f4bb-b404-433d-a80b-2ce7e5bed685","Type":"ContainerDied","Data":"f5167823de44819f829ccc24c060118427debd3d3d425fd4d801ab49eee6c570"}
Sep 18 15:50:46.008426 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:46.008347    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s" event={"ID":"cd72f4bb-b404-433d-a80b-2ce7e5bed685","Type":"ContainerStarted","Data":"d00ee6bcda1c19fa6d3960e75aca7dd9d33497cfcd27eddfbc6aa605bb837b1e"}
Sep 18 15:50:46.019632 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:46.019610    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:50:46.566459 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:46.566430    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:47.147961 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:47.147934    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s"
Sep 18 15:50:47.287990 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:47.287905    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd72f4bb-b404-433d-a80b-2ce7e5bed685-maas-api-service-ca\") pod \"cd72f4bb-b404-433d-a80b-2ce7e5bed685\" (UID: \"cd72f4bb-b404-433d-a80b-2ce7e5bed685\") "
Sep 18 15:50:47.288233 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:47.288211    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd72f4bb-b404-433d-a80b-2ce7e5bed685-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd72f4bb-b404-433d-a80b-2ce7e5bed685" (UID: "cd72f4bb-b404-433d-a80b-2ce7e5bed685"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:50:47.389269 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:47.389232    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd72f4bb-b404-433d-a80b-2ce7e5bed685-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:47.412157 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:47.412124    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:48.019955 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:48.019848    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s"
Sep 18 15:50:48.019955 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:48.019925    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s" event={"ID":"cd72f4bb-b404-433d-a80b-2ce7e5bed685","Type":"ContainerDied","Data":"d00ee6bcda1c19fa6d3960e75aca7dd9d33497cfcd27eddfbc6aa605bb837b1e"}
Sep 18 15:50:48.020406 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:48.019961    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d00ee6bcda1c19fa6d3960e75aca7dd9d33497cfcd27eddfbc6aa605bb837b1e"
Sep 18 15:50:48.186733 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:48.186691    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:48.868553 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:48.868509    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:49.587333 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:49.587277    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:56.095814 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:56.095736    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:56.794456 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:56.794422    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:56.838244 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:56.838210    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"]
Sep 18 15:50:56.838489 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:56.838463    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs" podUID="1da41811-bbc6-4bf9-ac26-89b143b20029" containerName="payload-processing" containerID="cri-o://a3ef493f74ea7514ebcd331178e93fdff65e3b7940352cf30cd065bf67235b64" gracePeriod=30
Sep 18 15:50:56.846486 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:56.846434    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"]
Sep 18 15:50:56.846686 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:56.846667    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx" podUID="194618f7-406f-492c-9e5e-146fa5f6d171" containerName="payload-pre-processing" containerID="cri-o://e9b430e041012c3d66ff5f6a35e267964c5265fbfe9da5e41af941c4a526328d" gracePeriod=30
Sep 18 15:50:56.889751 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:56.889695    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:56.994744 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:56.994706    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:57.057357 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.057300    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s"]
Sep 18 15:50:57.059568 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.059544    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c9ad8682-kbs8s"]
Sep 18 15:50:57.070522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.070492    2559 generic.go:356] "Generic (PLEG): container finished" podID="194618f7-406f-492c-9e5e-146fa5f6d171" containerID="e9b430e041012c3d66ff5f6a35e267964c5265fbfe9da5e41af941c4a526328d" exitCode=0
Sep 18 15:50:57.070653 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.070569    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx" event={"ID":"194618f7-406f-492c-9e5e-146fa5f6d171","Type":"ContainerDied","Data":"e9b430e041012c3d66ff5f6a35e267964c5265fbfe9da5e41af941c4a526328d"}
Sep 18 15:50:57.072450 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.072421    2559 generic.go:356] "Generic (PLEG): container finished" podID="1da41811-bbc6-4bf9-ac26-89b143b20029" containerID="a3ef493f74ea7514ebcd331178e93fdff65e3b7940352cf30cd065bf67235b64" exitCode=0
Sep 18 15:50:57.072607 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.072451    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs" event={"ID":"1da41811-bbc6-4bf9-ac26-89b143b20029","Type":"ContainerDied","Data":"a3ef493f74ea7514ebcd331178e93fdff65e3b7940352cf30cd065bf67235b64"}
Sep 18 15:50:57.073376 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.073348    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:57.119806 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.119785    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"
Sep 18 15:50:57.122656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.122639    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"
Sep 18 15:50:57.263200 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.263160    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:57.281275 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.281246    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194618f7-406f-492c-9e5e-146fa5f6d171-plugins-config-volume\") pod \"194618f7-406f-492c-9e5e-146fa5f6d171\" (UID: \"194618f7-406f-492c-9e5e-146fa5f6d171\") "
Sep 18 15:50:57.281411 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.281321    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9slt\" (UniqueName: \"kubernetes.io/projected/1da41811-bbc6-4bf9-ac26-89b143b20029-kube-api-access-h9slt\") pod \"1da41811-bbc6-4bf9-ac26-89b143b20029\" (UID: \"1da41811-bbc6-4bf9-ac26-89b143b20029\") "
Sep 18 15:50:57.281411 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.281388    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf5qk\" (UniqueName: \"kubernetes.io/projected/194618f7-406f-492c-9e5e-146fa5f6d171-kube-api-access-pf5qk\") pod \"194618f7-406f-492c-9e5e-146fa5f6d171\" (UID: \"194618f7-406f-492c-9e5e-146fa5f6d171\") "
Sep 18 15:50:57.281498 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.281409    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da41811-bbc6-4bf9-ac26-89b143b20029-plugins-config-volume\") pod \"1da41811-bbc6-4bf9-ac26-89b143b20029\" (UID: \"1da41811-bbc6-4bf9-ac26-89b143b20029\") "
Sep 18 15:50:57.281611 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.281591    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194618f7-406f-492c-9e5e-146fa5f6d171-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "194618f7-406f-492c-9e5e-146fa5f6d171" (UID: "194618f7-406f-492c-9e5e-146fa5f6d171"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:50:57.281806 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.281783    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da41811-bbc6-4bf9-ac26-89b143b20029-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1da41811-bbc6-4bf9-ac26-89b143b20029" (UID: "1da41811-bbc6-4bf9-ac26-89b143b20029"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:50:57.283457 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.283437    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da41811-bbc6-4bf9-ac26-89b143b20029-kube-api-access-h9slt" (OuterVolumeSpecName: "kube-api-access-h9slt") pod "1da41811-bbc6-4bf9-ac26-89b143b20029" (UID: "1da41811-bbc6-4bf9-ac26-89b143b20029"). InnerVolumeSpecName "kube-api-access-h9slt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:50:57.283528 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.283514    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194618f7-406f-492c-9e5e-146fa5f6d171-kube-api-access-pf5qk" (OuterVolumeSpecName: "kube-api-access-pf5qk") pod "194618f7-406f-492c-9e5e-146fa5f6d171" (UID: "194618f7-406f-492c-9e5e-146fa5f6d171"). InnerVolumeSpecName "kube-api-access-pf5qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:50:57.383016 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.382938    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9slt\" (UniqueName: \"kubernetes.io/projected/1da41811-bbc6-4bf9-ac26-89b143b20029-kube-api-access-h9slt\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:57.383016 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.382966    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf5qk\" (UniqueName: \"kubernetes.io/projected/194618f7-406f-492c-9e5e-146fa5f6d171-kube-api-access-pf5qk\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:57.383016 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.382975    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da41811-bbc6-4bf9-ac26-89b143b20029-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:57.383016 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.382985    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194618f7-406f-492c-9e5e-146fa5f6d171-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:50:57.509395 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.509357    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:57.744368 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.744335    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd72f4bb-b404-433d-a80b-2ce7e5bed685" path="/var/lib/kubelet/pods/cd72f4bb-b404-433d-a80b-2ce7e5bed685/volumes"
Sep 18 15:50:57.905452 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:57.905420    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:58.001668 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:58.001580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:58.078918 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:58.078889    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"
Sep 18 15:50:58.079081 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:58.078914    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs" event={"ID":"1da41811-bbc6-4bf9-ac26-89b143b20029","Type":"ContainerDied","Data":"62ab192eb3483b6d7b58a30df18fdddc1b33ad2cc782a37377c21661296e9520"}
Sep 18 15:50:58.079081 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:58.078976    2559 scope.go:117] "RemoveContainer" containerID="a3ef493f74ea7514ebcd331178e93fdff65e3b7940352cf30cd065bf67235b64"
Sep 18 15:50:58.080487 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:58.080466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx" event={"ID":"194618f7-406f-492c-9e5e-146fa5f6d171","Type":"ContainerDied","Data":"3b5b4fdc1ca68068f0726cd8e14df6309fea52ee29fd5dd6c9277a70830ce587"}
Sep 18 15:50:58.080545 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:58.080504    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"
Sep 18 15:50:58.088757 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:58.088738    2559 scope.go:117] "RemoveContainer" containerID="e9b430e041012c3d66ff5f6a35e267964c5265fbfe9da5e41af941c4a526328d"
Sep 18 15:50:58.098615 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:58.098593    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"]
Sep 18 15:50:58.102580 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:58.102558    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:58.105281 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:58.105263    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9ad8682-c69b5b85d-xlbwx"]
Sep 18 15:50:58.121579 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:58.121555    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"]
Sep 18 15:50:58.128231 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:58.128204    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9ad8682-6697557549-8jmxs"]
Sep 18 15:50:58.420123 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:58.420029    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:59.380155 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:59.380120    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:59.483514 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:59.483477    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:50:59.743520 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:59.743480    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="194618f7-406f-492c-9e5e-146fa5f6d171" path="/var/lib/kubelet/pods/194618f7-406f-492c-9e5e-146fa5f6d171/volumes"
Sep 18 15:50:59.743943 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:50:59.743922    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da41811-bbc6-4bf9-ac26-89b143b20029" path="/var/lib/kubelet/pods/1da41811-bbc6-4bf9-ac26-89b143b20029/volumes"
Sep 18 15:51:00.237226 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:00.237187    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:00.917662 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:00.917622    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:06.293453 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:06.293417    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:06.575751 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:06.575657    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:06.679937 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:06.679905    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:07.003872 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:07.003821    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:08.483638 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:08.483606    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:51:16.789188 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:16.789101    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:22.280054 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:22.280023    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:22.381452 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:22.381421    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:32.931730 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:32.931690    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:39.144325 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:39.144287    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:39.208580 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:39.208549    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:51:39.259795 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:39.259752    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:39.357379 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:39.357345    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:48.354622 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:48.354586    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:49.006787 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.006754    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"]
Sep 18 15:51:49.007166 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.007153    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="194618f7-406f-492c-9e5e-146fa5f6d171" containerName="payload-pre-processing"
Sep 18 15:51:49.007213 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.007168    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="194618f7-406f-492c-9e5e-146fa5f6d171" containerName="payload-pre-processing"
Sep 18 15:51:49.007213 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.007182    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da41811-bbc6-4bf9-ac26-89b143b20029" containerName="payload-processing"
Sep 18 15:51:49.007213 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.007188    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da41811-bbc6-4bf9-ac26-89b143b20029" containerName="payload-processing"
Sep 18 15:51:49.007213 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.007197    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd72f4bb-b404-433d-a80b-2ce7e5bed685" containerName="revoke-keys"
Sep 18 15:51:49.007213 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.007203    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd72f4bb-b404-433d-a80b-2ce7e5bed685" containerName="revoke-keys"
Sep 18 15:51:49.007369 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.007272    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da41811-bbc6-4bf9-ac26-89b143b20029" containerName="payload-processing"
Sep 18 15:51:49.007369 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.007279    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd72f4bb-b404-433d-a80b-2ce7e5bed685" containerName="revoke-keys"
Sep 18 15:51:49.007369 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.007286    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="194618f7-406f-492c-9e5e-146fa5f6d171" containerName="payload-pre-processing"
Sep 18 15:51:49.011977 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.011960    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.016926 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.016904    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9792ff\"/\"distinct-2-w3-9792ff-kserve-self-signed-certs\""
Sep 18 15:51:49.039182 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.039152    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:49.047942 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.047921    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"]
Sep 18 15:51:49.154653 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.154617    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-home\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.154653 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.154663    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ddt4\" (UniqueName: \"kubernetes.io/projected/a59feb5c-d29c-4281-a2ba-afd314a237d0-kube-api-access-9ddt4\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.154902 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.154766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-dshm\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.154902 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.154790    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-model-cache\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.154902 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.154814    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59feb5c-d29c-4281-a2ba-afd314a237d0-tls-certs\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.155043 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.154957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-tmp-dir\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.255993 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.255958    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-home\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.256185 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.256002    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ddt4\" (UniqueName: \"kubernetes.io/projected/a59feb5c-d29c-4281-a2ba-afd314a237d0-kube-api-access-9ddt4\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.256185 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.256081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-dshm\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.256310 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.256199    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-model-cache\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.256310 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.256238    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59feb5c-d29c-4281-a2ba-afd314a237d0-tls-certs\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.256310 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.256276    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-tmp-dir\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.256441 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.256408    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-home\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.256611 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.256583    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-tmp-dir\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.256611 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.256606    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-model-cache\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.258433 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.258387    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-dshm\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.258576 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.258558    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59feb5c-d29c-4281-a2ba-afd314a237d0-tls-certs\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.264343 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.264311    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ddt4\" (UniqueName: \"kubernetes.io/projected/a59feb5c-d29c-4281-a2ba-afd314a237d0-kube-api-access-9ddt4\") pod \"distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") " pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.321224 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.321189    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:49.591600 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:49.591576    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"]
Sep 18 15:51:49.593905 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:51:49.593878    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda59feb5c_d29c_4281_a2ba_afd314a237d0.slice/crio-25ca155000c219292b49f8414a9e6720fa1b41c9642b404ab20f818cfec39082 WatchSource:0}: Error finding container 25ca155000c219292b49f8414a9e6720fa1b41c9642b404ab20f818cfec39082: Status 404 returned error can't find the container with id 25ca155000c219292b49f8414a9e6720fa1b41c9642b404ab20f818cfec39082
Sep 18 15:51:50.350527 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:50.350488    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h" event={"ID":"a59feb5c-d29c-4281-a2ba-afd314a237d0","Type":"ContainerStarted","Data":"b6288d13088e3a9f6295bbf201636c31625206765305256826ead8b6647908f0"}
Sep 18 15:51:50.350527 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:50.350534    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h" event={"ID":"a59feb5c-d29c-4281-a2ba-afd314a237d0","Type":"ContainerStarted","Data":"25ca155000c219292b49f8414a9e6720fa1b41c9642b404ab20f818cfec39082"}
Sep 18 15:51:50.372714 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:50.372657    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h" podStartSLOduration=2.372640579 podStartE2EDuration="2.372640579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:51:50.371103315 +0000 UTC m=+1032.932335456" watchObservedRunningTime="2026-09-18 15:51:50.372640579 +0000 UTC m=+1032.933872719"
Sep 18 15:51:55.159108 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:55.159072    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:51:59.321696 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:59.321665    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:59.322127 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:59.321724    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:59.334095 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:59.334072    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:51:59.406703 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:51:59.406676    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:52:01.061323 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:01.061287    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:01.140728 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:01.140694    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:01.247423 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:01.247378    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:05.496495 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:05.496439    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:07.470801 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:07.470761    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:52:11.723697 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:11.723654    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:12.475581 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:12.475540    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:15.311930 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:15.311898    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"]
Sep 18 15:52:15.334490 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:15.334460    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:19.013093 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:19.013058    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:19.123707 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:19.123677    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:19.219543 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:19.219489    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:19.319339 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:19.319227    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:19.835784 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:19.835752    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"]
Sep 18 15:52:19.857477 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:19.857440    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:19.982985 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:19.982950    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:20.054965 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:20.054926    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj"]
Sep 18 15:52:20.058762 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:20.058740    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj"
Sep 18 15:52:20.062495 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:20.062472    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 15:52:20.062744 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:20.062663    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 15:52:20.077869 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:20.077830    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj"]
Sep 18 15:52:20.131829 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:20.131733    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:20.148000 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:20.147975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab66cfd-4fd3-483a-8677-fad56e4b3f80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj\" (UID: \"cab66cfd-4fd3-483a-8677-fad56e4b3f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj"
Sep 18 15:52:20.228328 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:20.228289    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:20.248602 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:20.248572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab66cfd-4fd3-483a-8677-fad56e4b3f80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj\" (UID: \"cab66cfd-4fd3-483a-8677-fad56e4b3f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj"
Sep 18 15:52:20.249221 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:20.249203    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab66cfd-4fd3-483a-8677-fad56e4b3f80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj\" (UID: \"cab66cfd-4fd3-483a-8677-fad56e4b3f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj"
Sep 18 15:52:20.370721 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:20.370691    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj"
Sep 18 15:52:20.643280 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:20.643255    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj"]
Sep 18 15:52:20.645074 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:52:20.645045    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab66cfd_4fd3_483a_8677_fad56e4b3f80.slice/crio-a1f69d535ebe41d912acb9a55b4d31e6179d106c53c82d3246ae0d89935bb27b WatchSource:0}: Error finding container a1f69d535ebe41d912acb9a55b4d31e6179d106c53c82d3246ae0d89935bb27b: Status 404 returned error can't find the container with id a1f69d535ebe41d912acb9a55b4d31e6179d106c53c82d3246ae0d89935bb27b
Sep 18 15:52:21.189023 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:21.188939    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:21.509786 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:21.509752    2559 generic.go:356] "Generic (PLEG): container finished" podID="cab66cfd-4fd3-483a-8677-fad56e4b3f80" containerID="50310bb37208d429af7f649a9c9cf6e9b061b79ed690fd88bf3e4a1a4ef8ef5b" exitCode=0
Sep 18 15:52:21.509988 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:21.509835    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj" event={"ID":"cab66cfd-4fd3-483a-8677-fad56e4b3f80","Type":"ContainerDied","Data":"50310bb37208d429af7f649a9c9cf6e9b061b79ed690fd88bf3e4a1a4ef8ef5b"}
Sep 18 15:52:21.509988 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:21.509906    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj" event={"ID":"cab66cfd-4fd3-483a-8677-fad56e4b3f80","Type":"ContainerStarted","Data":"a1f69d535ebe41d912acb9a55b4d31e6179d106c53c82d3246ae0d89935bb27b"}
Sep 18 15:52:21.624479 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:21.624372    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:21.914462 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:21.914376    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:22.622739 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:22.622703    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:22.656102 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:22.656072    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj"
Sep 18 15:52:22.773984 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:22.773890    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab66cfd-4fd3-483a-8677-fad56e4b3f80-maas-api-service-ca\") pod \"cab66cfd-4fd3-483a-8677-fad56e4b3f80\" (UID: \"cab66cfd-4fd3-483a-8677-fad56e4b3f80\") "
Sep 18 15:52:22.774283 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:22.774260    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab66cfd-4fd3-483a-8677-fad56e4b3f80-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cab66cfd-4fd3-483a-8677-fad56e4b3f80" (UID: "cab66cfd-4fd3-483a-8677-fad56e4b3f80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:52:22.875656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:22.875619    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab66cfd-4fd3-483a-8677-fad56e4b3f80-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:23.421677 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:23.421640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:23.520522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:23.520488    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj" event={"ID":"cab66cfd-4fd3-483a-8677-fad56e4b3f80","Type":"ContainerDied","Data":"a1f69d535ebe41d912acb9a55b4d31e6179d106c53c82d3246ae0d89935bb27b"}
Sep 18 15:52:23.520522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:23.520515    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj"
Sep 18 15:52:23.520720 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:23.520540    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1f69d535ebe41d912acb9a55b4d31e6179d106c53c82d3246ae0d89935bb27b"
Sep 18 15:52:24.165153 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:24.165110    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:27.471575 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:27.471531    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:27.577299 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:27.577259    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:28.158141 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:28.158108    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:30.331930 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.331893    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g" podUID="75e9b9a9-1a1e-4428-8653-a994ac150d09" containerName="main" containerID="cri-o://6cbe22ed024b1f1deddd81ff6e67ae5a8ff5620d1310b25c1b153a84eb9c5e85" gracePeriod=45
Sep 18 15:52:30.471084 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.471045    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:52:30.558191 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.558162    2559 generic.go:356] "Generic (PLEG): container finished" podID="75e9b9a9-1a1e-4428-8653-a994ac150d09" containerID="6cbe22ed024b1f1deddd81ff6e67ae5a8ff5620d1310b25c1b153a84eb9c5e85" exitCode=0
Sep 18 15:52:30.558370 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.558242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g" event={"ID":"75e9b9a9-1a1e-4428-8653-a994ac150d09","Type":"ContainerDied","Data":"6cbe22ed024b1f1deddd81ff6e67ae5a8ff5620d1310b25c1b153a84eb9c5e85"}
Sep 18 15:52:30.587325 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.587253    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:52:30.658234 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.658202    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:30.753127 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.753096    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-model-cache\") pod \"75e9b9a9-1a1e-4428-8653-a994ac150d09\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") "
Sep 18 15:52:30.753321 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.753170    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-tmp-dir\") pod \"75e9b9a9-1a1e-4428-8653-a994ac150d09\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") "
Sep 18 15:52:30.753321 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.753214    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-home\") pod \"75e9b9a9-1a1e-4428-8653-a994ac150d09\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") "
Sep 18 15:52:30.753321 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.753235    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ghfj\" (UniqueName: \"kubernetes.io/projected/75e9b9a9-1a1e-4428-8653-a994ac150d09-kube-api-access-7ghfj\") pod \"75e9b9a9-1a1e-4428-8653-a994ac150d09\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") "
Sep 18 15:52:30.753321 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.753266    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e9b9a9-1a1e-4428-8653-a994ac150d09-tls-certs\") pod \"75e9b9a9-1a1e-4428-8653-a994ac150d09\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") "
Sep 18 15:52:30.753321 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.753305    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-dshm\") pod \"75e9b9a9-1a1e-4428-8653-a994ac150d09\" (UID: \"75e9b9a9-1a1e-4428-8653-a994ac150d09\") "
Sep 18 15:52:30.753586 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.753459    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-model-cache" (OuterVolumeSpecName: "model-cache") pod "75e9b9a9-1a1e-4428-8653-a994ac150d09" (UID: "75e9b9a9-1a1e-4428-8653-a994ac150d09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:52:30.753586 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.753484    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75e9b9a9-1a1e-4428-8653-a994ac150d09" (UID: "75e9b9a9-1a1e-4428-8653-a994ac150d09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:52:30.753586 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.753506    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-home" (OuterVolumeSpecName: "home") pod "75e9b9a9-1a1e-4428-8653-a994ac150d09" (UID: "75e9b9a9-1a1e-4428-8653-a994ac150d09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:52:30.753706 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.753676    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-model-cache\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:30.753706 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.753692    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-tmp-dir\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:30.753706 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.753700    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-home\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:30.755563 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.755528    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-dshm" (OuterVolumeSpecName: "dshm") pod "75e9b9a9-1a1e-4428-8653-a994ac150d09" (UID: "75e9b9a9-1a1e-4428-8653-a994ac150d09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:52:30.755679 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.755552    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e9b9a9-1a1e-4428-8653-a994ac150d09-kube-api-access-7ghfj" (OuterVolumeSpecName: "kube-api-access-7ghfj") pod "75e9b9a9-1a1e-4428-8653-a994ac150d09" (UID: "75e9b9a9-1a1e-4428-8653-a994ac150d09"). InnerVolumeSpecName "kube-api-access-7ghfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:52:30.755679 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.755566    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e9b9a9-1a1e-4428-8653-a994ac150d09-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75e9b9a9-1a1e-4428-8653-a994ac150d09" (UID: "75e9b9a9-1a1e-4428-8653-a994ac150d09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:52:30.860979 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.855294    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ghfj\" (UniqueName: \"kubernetes.io/projected/75e9b9a9-1a1e-4428-8653-a994ac150d09-kube-api-access-7ghfj\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:30.860979 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.855343    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e9b9a9-1a1e-4428-8653-a994ac150d09-tls-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:30.860979 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:30.855357    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e9b9a9-1a1e-4428-8653-a994ac150d09-dshm\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:31.250724 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.250687    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:31.282880 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.282837    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"]
Sep 18 15:52:31.283101 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.283074    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz" podUID="f2cdb485-d211-4db7-9c80-0b8a3db81039" containerName="payload-processing" containerID="cri-o://4337b80315ef21ce78f4b18180982dd249913ff86a09c51a5fdf4b118f6d11d4" gracePeriod=30
Sep 18 15:52:31.291722 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.291695    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"]
Sep 18 15:52:31.292137 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.291915    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb" podUID="8d83550d-db6b-4d86-9df9-bcea236c32b1" containerName="payload-pre-processing" containerID="cri-o://94cd1a33b0740dc4e7c0267789649bb4bc70c139477860f4aba5a82315b01d3f" gracePeriod=30
Sep 18 15:52:31.329356 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.329324    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:31.434886 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.434826    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:31.498695 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.498662    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj"]
Sep 18 15:52:31.501510 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.501452    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a1f773-9dtzj"]
Sep 18 15:52:31.508622 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.508593    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:31.565336 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.565312    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"
Sep 18 15:52:31.566971 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.566939    2559 generic.go:356] "Generic (PLEG): container finished" podID="f2cdb485-d211-4db7-9c80-0b8a3db81039" containerID="4337b80315ef21ce78f4b18180982dd249913ff86a09c51a5fdf4b118f6d11d4" exitCode=0
Sep 18 15:52:31.567073 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.566973    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz" event={"ID":"f2cdb485-d211-4db7-9c80-0b8a3db81039","Type":"ContainerDied","Data":"4337b80315ef21ce78f4b18180982dd249913ff86a09c51a5fdf4b118f6d11d4"}
Sep 18 15:52:31.567073 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.566989    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"
Sep 18 15:52:31.567073 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.567023    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz" event={"ID":"f2cdb485-d211-4db7-9c80-0b8a3db81039","Type":"ContainerDied","Data":"3a0f9ead60c899d2b5c31b210744a22dd2eec792a1db2d9ab17690e4f5373fd2"}
Sep 18 15:52:31.567073 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.567055    2559 scope.go:117] "RemoveContainer" containerID="4337b80315ef21ce78f4b18180982dd249913ff86a09c51a5fdf4b118f6d11d4"
Sep 18 15:52:31.568496 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.568478    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"
Sep 18 15:52:31.568618 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.568600    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"
Sep 18 15:52:31.568676 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.568617    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g" event={"ID":"75e9b9a9-1a1e-4428-8653-a994ac150d09","Type":"ContainerDied","Data":"67522ce9f54cd38073678f5a1a0a8a6de67235e1af1151fa08b0bad8382fef94"}
Sep 18 15:52:31.570198 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.570175    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d83550d-db6b-4d86-9df9-bcea236c32b1" containerID="94cd1a33b0740dc4e7c0267789649bb4bc70c139477860f4aba5a82315b01d3f" exitCode=0
Sep 18 15:52:31.570305 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.570212    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"
Sep 18 15:52:31.570305 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.570241    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb" event={"ID":"8d83550d-db6b-4d86-9df9-bcea236c32b1","Type":"ContainerDied","Data":"94cd1a33b0740dc4e7c0267789649bb4bc70c139477860f4aba5a82315b01d3f"}
Sep 18 15:52:31.570305 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.570277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb" event={"ID":"8d83550d-db6b-4d86-9df9-bcea236c32b1","Type":"ContainerDied","Data":"cab7a7e08cdd2fbfb198547f7d1a9bcfa4447df12f13a4c57dc388338d0796b9"}
Sep 18 15:52:31.576583 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.576563    2559 scope.go:117] "RemoveContainer" containerID="4337b80315ef21ce78f4b18180982dd249913ff86a09c51a5fdf4b118f6d11d4"
Sep 18 15:52:31.576899 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:52:31.576876    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4337b80315ef21ce78f4b18180982dd249913ff86a09c51a5fdf4b118f6d11d4\": container with ID starting with 4337b80315ef21ce78f4b18180982dd249913ff86a09c51a5fdf4b118f6d11d4 not found: ID does not exist" containerID="4337b80315ef21ce78f4b18180982dd249913ff86a09c51a5fdf4b118f6d11d4"
Sep 18 15:52:31.576965 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.576908    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4337b80315ef21ce78f4b18180982dd249913ff86a09c51a5fdf4b118f6d11d4"} err="failed to get container status \"4337b80315ef21ce78f4b18180982dd249913ff86a09c51a5fdf4b118f6d11d4\": rpc error: code = NotFound desc = could not find container \"4337b80315ef21ce78f4b18180982dd249913ff86a09c51a5fdf4b118f6d11d4\": container with ID starting with 4337b80315ef21ce78f4b18180982dd249913ff86a09c51a5fdf4b118f6d11d4 not found: ID does not exist"
Sep 18 15:52:31.576965 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.576926    2559 scope.go:117] "RemoveContainer" containerID="6cbe22ed024b1f1deddd81ff6e67ae5a8ff5620d1310b25c1b153a84eb9c5e85"
Sep 18 15:52:31.584689 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.584669    2559 scope.go:117] "RemoveContainer" containerID="94cd1a33b0740dc4e7c0267789649bb4bc70c139477860f4aba5a82315b01d3f"
Sep 18 15:52:31.594328 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.594311    2559 scope.go:117] "RemoveContainer" containerID="94cd1a33b0740dc4e7c0267789649bb4bc70c139477860f4aba5a82315b01d3f"
Sep 18 15:52:31.594612 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:52:31.594594    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94cd1a33b0740dc4e7c0267789649bb4bc70c139477860f4aba5a82315b01d3f\": container with ID starting with 94cd1a33b0740dc4e7c0267789649bb4bc70c139477860f4aba5a82315b01d3f not found: ID does not exist" containerID="94cd1a33b0740dc4e7c0267789649bb4bc70c139477860f4aba5a82315b01d3f"
Sep 18 15:52:31.594671 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.594619    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94cd1a33b0740dc4e7c0267789649bb4bc70c139477860f4aba5a82315b01d3f"} err="failed to get container status \"94cd1a33b0740dc4e7c0267789649bb4bc70c139477860f4aba5a82315b01d3f\": rpc error: code = NotFound desc = could not find container \"94cd1a33b0740dc4e7c0267789649bb4bc70c139477860f4aba5a82315b01d3f\": container with ID starting with 94cd1a33b0740dc4e7c0267789649bb4bc70c139477860f4aba5a82315b01d3f not found: ID does not exist"
Sep 18 15:52:31.619120 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.619096    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"]
Sep 18 15:52:31.622886 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.622850    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a1f773/test-model-a1f773-kserve-74ddf875b7-wp86g"]
Sep 18 15:52:31.662441 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.662413    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2cdb485-d211-4db7-9c80-0b8a3db81039-plugins-config-volume\") pod \"f2cdb485-d211-4db7-9c80-0b8a3db81039\" (UID: \"f2cdb485-d211-4db7-9c80-0b8a3db81039\") "
Sep 18 15:52:31.662562 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.662523    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rntj\" (UniqueName: \"kubernetes.io/projected/f2cdb485-d211-4db7-9c80-0b8a3db81039-kube-api-access-9rntj\") pod \"f2cdb485-d211-4db7-9c80-0b8a3db81039\" (UID: \"f2cdb485-d211-4db7-9c80-0b8a3db81039\") "
Sep 18 15:52:31.662906 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.662846    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cdb485-d211-4db7-9c80-0b8a3db81039-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2cdb485-d211-4db7-9c80-0b8a3db81039" (UID: "f2cdb485-d211-4db7-9c80-0b8a3db81039"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:52:31.664621 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.664599    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cdb485-d211-4db7-9c80-0b8a3db81039-kube-api-access-9rntj" (OuterVolumeSpecName: "kube-api-access-9rntj") pod "f2cdb485-d211-4db7-9c80-0b8a3db81039" (UID: "f2cdb485-d211-4db7-9c80-0b8a3db81039"). InnerVolumeSpecName "kube-api-access-9rntj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:52:31.745620 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.745590    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75e9b9a9-1a1e-4428-8653-a994ac150d09" path="/var/lib/kubelet/pods/75e9b9a9-1a1e-4428-8653-a994ac150d09/volumes"
Sep 18 15:52:31.745975 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.745960    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cab66cfd-4fd3-483a-8677-fad56e4b3f80" path="/var/lib/kubelet/pods/cab66cfd-4fd3-483a-8677-fad56e4b3f80/volumes"
Sep 18 15:52:31.763867 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.763829    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d83550d-db6b-4d86-9df9-bcea236c32b1-plugins-config-volume\") pod \"8d83550d-db6b-4d86-9df9-bcea236c32b1\" (UID: \"8d83550d-db6b-4d86-9df9-bcea236c32b1\") "
Sep 18 15:52:31.763971 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.763891    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpgrv\" (UniqueName: \"kubernetes.io/projected/8d83550d-db6b-4d86-9df9-bcea236c32b1-kube-api-access-qpgrv\") pod \"8d83550d-db6b-4d86-9df9-bcea236c32b1\" (UID: \"8d83550d-db6b-4d86-9df9-bcea236c32b1\") "
Sep 18 15:52:31.764146 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.764129    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rntj\" (UniqueName: \"kubernetes.io/projected/f2cdb485-d211-4db7-9c80-0b8a3db81039-kube-api-access-9rntj\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:31.764207 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.764152    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2cdb485-d211-4db7-9c80-0b8a3db81039-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:31.764253 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.764208    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d83550d-db6b-4d86-9df9-bcea236c32b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d83550d-db6b-4d86-9df9-bcea236c32b1" (UID: "8d83550d-db6b-4d86-9df9-bcea236c32b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:52:31.765820 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.765798    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d83550d-db6b-4d86-9df9-bcea236c32b1-kube-api-access-qpgrv" (OuterVolumeSpecName: "kube-api-access-qpgrv") pod "8d83550d-db6b-4d86-9df9-bcea236c32b1" (UID: "8d83550d-db6b-4d86-9df9-bcea236c32b1"). InnerVolumeSpecName "kube-api-access-qpgrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:52:31.852267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.852233    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:31.864938 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.864918    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d83550d-db6b-4d86-9df9-bcea236c32b1-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:31.864938 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.864939    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpgrv\" (UniqueName: \"kubernetes.io/projected/8d83550d-db6b-4d86-9df9-bcea236c32b1-kube-api-access-qpgrv\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:31.884491 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.884469    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"]
Sep 18 15:52:31.888237 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.888220    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a1f773-745c674775-6ksnz"]
Sep 18 15:52:31.900380 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.900361    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"]
Sep 18 15:52:31.906740 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:31.906720    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a1f773-6685bf8cd-g52bb"]
Sep 18 15:52:32.348687 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:32.348655    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:33.104430 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:33.104399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:33.744026 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:33.743990    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d83550d-db6b-4d86-9df9-bcea236c32b1" path="/var/lib/kubelet/pods/8d83550d-db6b-4d86-9df9-bcea236c32b1/volumes"
Sep 18 15:52:33.744363 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:33.744351    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2cdb485-d211-4db7-9c80-0b8a3db81039" path="/var/lib/kubelet/pods/f2cdb485-d211-4db7-9c80-0b8a3db81039/volumes"
Sep 18 15:52:34.111687 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:34.111596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:34.860945 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:34.860903    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4" podUID="c8be15b0-a494-4ff4-a6e2-98048350a522" containerName="main" containerID="cri-o://7ec4399364995cc5446926966be3803c134006aba72cbefe2260cf53895efaf2" gracePeriod=45
Sep 18 15:52:35.013505 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.013474    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:35.096147 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.096115    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:35.114533 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.114480    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:52:35.191764 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.191732    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rx6l\" (UniqueName: \"kubernetes.io/projected/c8be15b0-a494-4ff4-a6e2-98048350a522-kube-api-access-7rx6l\") pod \"c8be15b0-a494-4ff4-a6e2-98048350a522\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") "
Sep 18 15:52:35.191955 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.191774    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-home\") pod \"c8be15b0-a494-4ff4-a6e2-98048350a522\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") "
Sep 18 15:52:35.191955 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.191798    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-tmp-dir\") pod \"c8be15b0-a494-4ff4-a6e2-98048350a522\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") "
Sep 18 15:52:35.191955 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.191814    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-dshm\") pod \"c8be15b0-a494-4ff4-a6e2-98048350a522\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") "
Sep 18 15:52:35.191955 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.191841    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8be15b0-a494-4ff4-a6e2-98048350a522-tls-certs\") pod \"c8be15b0-a494-4ff4-a6e2-98048350a522\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") "
Sep 18 15:52:35.191955 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.191926    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-model-cache\") pod \"c8be15b0-a494-4ff4-a6e2-98048350a522\" (UID: \"c8be15b0-a494-4ff4-a6e2-98048350a522\") "
Sep 18 15:52:35.192236 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.192058    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-home" (OuterVolumeSpecName: "home") pod "c8be15b0-a494-4ff4-a6e2-98048350a522" (UID: "c8be15b0-a494-4ff4-a6e2-98048350a522"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:52:35.192236 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.192079    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8be15b0-a494-4ff4-a6e2-98048350a522" (UID: "c8be15b0-a494-4ff4-a6e2-98048350a522"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:52:35.192339 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.192234    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8be15b0-a494-4ff4-a6e2-98048350a522" (UID: "c8be15b0-a494-4ff4-a6e2-98048350a522"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:52:35.192339 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.192308    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-home\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:35.192339 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.192327    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-tmp-dir\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:35.194132 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.194108    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8be15b0-a494-4ff4-a6e2-98048350a522-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8be15b0-a494-4ff4-a6e2-98048350a522" (UID: "c8be15b0-a494-4ff4-a6e2-98048350a522"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:52:35.194429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.194398    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-dshm" (OuterVolumeSpecName: "dshm") pod "c8be15b0-a494-4ff4-a6e2-98048350a522" (UID: "c8be15b0-a494-4ff4-a6e2-98048350a522"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:52:35.194429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.194414    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8be15b0-a494-4ff4-a6e2-98048350a522-kube-api-access-7rx6l" (OuterVolumeSpecName: "kube-api-access-7rx6l") pod "c8be15b0-a494-4ff4-a6e2-98048350a522" (UID: "c8be15b0-a494-4ff4-a6e2-98048350a522"). InnerVolumeSpecName "kube-api-access-7rx6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:52:35.242359 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.242330    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:35.293364 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.293340    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-model-cache\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:35.293364 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.293362    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rx6l\" (UniqueName: \"kubernetes.io/projected/c8be15b0-a494-4ff4-a6e2-98048350a522-kube-api-access-7rx6l\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:35.293505 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.293373    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8be15b0-a494-4ff4-a6e2-98048350a522-dshm\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:35.293505 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.293382    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8be15b0-a494-4ff4-a6e2-98048350a522-tls-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:35.602058 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.602000    2559 generic.go:356] "Generic (PLEG): container finished" podID="c8be15b0-a494-4ff4-a6e2-98048350a522" containerID="7ec4399364995cc5446926966be3803c134006aba72cbefe2260cf53895efaf2" exitCode=0
Sep 18 15:52:35.602239 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.602065    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"
Sep 18 15:52:35.602239 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.602088    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4" event={"ID":"c8be15b0-a494-4ff4-a6e2-98048350a522","Type":"ContainerDied","Data":"7ec4399364995cc5446926966be3803c134006aba72cbefe2260cf53895efaf2"}
Sep 18 15:52:35.602239 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.602131    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4" event={"ID":"c8be15b0-a494-4ff4-a6e2-98048350a522","Type":"ContainerDied","Data":"2e853e0f81a2b58741843f9cccc4dc14b39c96a55b6234c512ec3a025fcefeeb"}
Sep 18 15:52:35.602239 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.602152    2559 scope.go:117] "RemoveContainer" containerID="7ec4399364995cc5446926966be3803c134006aba72cbefe2260cf53895efaf2"
Sep 18 15:52:35.612122 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.612102    2559 scope.go:117] "RemoveContainer" containerID="7ec4399364995cc5446926966be3803c134006aba72cbefe2260cf53895efaf2"
Sep 18 15:52:35.612417 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:52:35.612398    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec4399364995cc5446926966be3803c134006aba72cbefe2260cf53895efaf2\": container with ID starting with 7ec4399364995cc5446926966be3803c134006aba72cbefe2260cf53895efaf2 not found: ID does not exist" containerID="7ec4399364995cc5446926966be3803c134006aba72cbefe2260cf53895efaf2"
Sep 18 15:52:35.612471 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.612426    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec4399364995cc5446926966be3803c134006aba72cbefe2260cf53895efaf2"} err="failed to get container status \"7ec4399364995cc5446926966be3803c134006aba72cbefe2260cf53895efaf2\": rpc error: code = NotFound desc = could not find container \"7ec4399364995cc5446926966be3803c134006aba72cbefe2260cf53895efaf2\": container with ID starting with 7ec4399364995cc5446926966be3803c134006aba72cbefe2260cf53895efaf2 not found: ID does not exist"
Sep 18 15:52:35.626757 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.626733    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"]
Sep 18 15:52:35.627112 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627098    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cdb485-d211-4db7-9c80-0b8a3db81039" containerName="payload-processing"
Sep 18 15:52:35.627159 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627114    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cdb485-d211-4db7-9c80-0b8a3db81039" containerName="payload-processing"
Sep 18 15:52:35.627159 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627134    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d83550d-db6b-4d86-9df9-bcea236c32b1" containerName="payload-pre-processing"
Sep 18 15:52:35.627159 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627140    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d83550d-db6b-4d86-9df9-bcea236c32b1" containerName="payload-pre-processing"
Sep 18 15:52:35.627159 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627147    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e9b9a9-1a1e-4428-8653-a994ac150d09" containerName="main"
Sep 18 15:52:35.627159 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627152    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e9b9a9-1a1e-4428-8653-a994ac150d09" containerName="main"
Sep 18 15:52:35.627159 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627159    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8be15b0-a494-4ff4-a6e2-98048350a522" containerName="main"
Sep 18 15:52:35.627380 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627165    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8be15b0-a494-4ff4-a6e2-98048350a522" containerName="main"
Sep 18 15:52:35.627380 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627172    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cab66cfd-4fd3-483a-8677-fad56e4b3f80" containerName="revoke-keys"
Sep 18 15:52:35.627380 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627178    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cab66cfd-4fd3-483a-8677-fad56e4b3f80" containerName="revoke-keys"
Sep 18 15:52:35.627380 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627240    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8be15b0-a494-4ff4-a6e2-98048350a522" containerName="main"
Sep 18 15:52:35.627380 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627247    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2cdb485-d211-4db7-9c80-0b8a3db81039" containerName="payload-processing"
Sep 18 15:52:35.627380 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627254    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e9b9a9-1a1e-4428-8653-a994ac150d09" containerName="main"
Sep 18 15:52:35.627380 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627259    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d83550d-db6b-4d86-9df9-bcea236c32b1" containerName="payload-pre-processing"
Sep 18 15:52:35.627380 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.627265    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cab66cfd-4fd3-483a-8677-fad56e4b3f80" containerName="revoke-keys"
Sep 18 15:52:35.631398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.631384    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.634234 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.634213    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f84020ef-openshift-default-dockercfg-jmgvm\""
Sep 18 15:52:35.641324 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.641301    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"]
Sep 18 15:52:35.651608 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.651587    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-70c6df/test-model-70c6df-kserve-d6b67888f-h6rb4"]
Sep 18 15:52:35.666484 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.666460    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"]
Sep 18 15:52:35.693065 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.693011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:35.696951 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.696924    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-workload-socket\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.697095 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.696960    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-envoy\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.697095 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.696992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpzsj\" (UniqueName: \"kubernetes.io/projected/b1bee9c4-37ea-4909-8e44-af4bd98fb252-kube-api-access-kpzsj\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.697193 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.697118    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-workload-certs\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.697229 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.697207    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-podinfo\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.697275 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.697259    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istiod-ca-cert\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.697325 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.697310    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-data\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.697369 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.697355    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-credential-socket\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.697408 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.697382    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-token\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.743647 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.743617    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8be15b0-a494-4ff4-a6e2-98048350a522" path="/var/lib/kubelet/pods/c8be15b0-a494-4ff4-a6e2-98048350a522/volumes"
Sep 18 15:52:35.798698 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.798669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istiod-ca-cert\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.798839 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.798703    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-data\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.798839 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.798727    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-credential-socket\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.798970 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.798841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-token\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.798970 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.798909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-workload-socket\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.798970 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.798940    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-envoy\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.799118 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.798972    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpzsj\" (UniqueName: \"kubernetes.io/projected/b1bee9c4-37ea-4909-8e44-af4bd98fb252-kube-api-access-kpzsj\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.799118 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.799088    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-data\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.799218 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.799169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-workload-certs\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.799218 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.799179    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-credential-socket\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.799332 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.799216    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-workload-socket\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.799332 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.799268    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-podinfo\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.799459 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.799441    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-workload-certs\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.799459 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.799451    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istiod-ca-cert\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.801345 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.801318    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-envoy\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.801733 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.801715    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-podinfo\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.808326 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.808308    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-token\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.812175 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.812160    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpzsj\" (UniqueName: \"kubernetes.io/projected/b1bee9c4-37ea-4909-8e44-af4bd98fb252-kube-api-access-kpzsj\") pod \"e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") " pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.941311 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.941285    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:35.970713 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:35.970677    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:36.239511 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:36.239450    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"]
Sep 18 15:52:36.242081 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:52:36.242057    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bee9c4_37ea_4909_8e44_af4bd98fb252.slice/crio-09942d22ae0a47b0c798b222a0b6e4298884b59e228b2c54cf1cd78dab561742 WatchSource:0}: Error finding container 09942d22ae0a47b0c798b222a0b6e4298884b59e228b2c54cf1cd78dab561742: Status 404 returned error can't find the container with id 09942d22ae0a47b0c798b222a0b6e4298884b59e228b2c54cf1cd78dab561742
Sep 18 15:52:36.244647 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:36.244617    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:52:36.244735 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:36.244695    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:52:36.244804 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:36.244743    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:52:36.610760 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:36.610662    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" event={"ID":"b1bee9c4-37ea-4909-8e44-af4bd98fb252","Type":"ContainerStarted","Data":"f2966897a5b54e63f77b765fb9152e2013f75e7c4e307d8335578004740dd816"}
Sep 18 15:52:36.610760 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:36.610730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" event={"ID":"b1bee9c4-37ea-4909-8e44-af4bd98fb252","Type":"ContainerStarted","Data":"09942d22ae0a47b0c798b222a0b6e4298884b59e228b2c54cf1cd78dab561742"}
Sep 18 15:52:36.650300 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:36.650250    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" podStartSLOduration=1.650236033 podStartE2EDuration="1.650236033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:52:36.648903037 +0000 UTC m=+1079.210135177" watchObservedRunningTime="2026-09-18 15:52:36.650236033 +0000 UTC m=+1079.211468211"
Sep 18 15:52:36.942232 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:36.942206    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:37.847325 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:37.847282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:37.947838 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:37.947806    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:38.456807 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:38.456772    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:38.574842 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:38.574807    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:38.621952 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:38.621920    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:38.623068 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:38.623046    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:52:38.689917 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:38.689886    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:38.778086 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:38.777996    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:39.041040 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:39.040954    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b"]
Sep 18 15:52:39.044529 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:39.044511    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b"
Sep 18 15:52:39.047406 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:39.047365    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 15:52:39.047560 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:39.047482    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 15:52:39.053494 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:39.053474    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b"]
Sep 18 15:52:39.132556 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:39.132527    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c758c09-d7db-404e-a5cb-5c17ec90e7fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b\" (UID: \"5c758c09-d7db-404e-a5cb-5c17ec90e7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b"
Sep 18 15:52:39.233098 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:39.233067    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c758c09-d7db-404e-a5cb-5c17ec90e7fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b\" (UID: \"5c758c09-d7db-404e-a5cb-5c17ec90e7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b"
Sep 18 15:52:39.233741 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:39.233719    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c758c09-d7db-404e-a5cb-5c17ec90e7fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b\" (UID: \"5c758c09-d7db-404e-a5cb-5c17ec90e7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b"
Sep 18 15:52:39.356790 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:39.356710    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b"
Sep 18 15:52:39.592142 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:39.592107    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:39.627522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:39.627499    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b"]
Sep 18 15:52:39.630165 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:52:39.630135    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c758c09_d7db_404e_a5cb_5c17ec90e7fb.slice/crio-1b4e2f667973f09d843d8ee7e6b23f755636298195ecf6ca71424deec36c75bc WatchSource:0}: Error finding container 1b4e2f667973f09d843d8ee7e6b23f755636298195ecf6ca71424deec36c75bc: Status 404 returned error can't find the container with id 1b4e2f667973f09d843d8ee7e6b23f755636298195ecf6ca71424deec36c75bc
Sep 18 15:52:40.575076 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:40.575033    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:40.634764 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:40.634685    2559 generic.go:356] "Generic (PLEG): container finished" podID="5c758c09-d7db-404e-a5cb-5c17ec90e7fb" containerID="3aa5f518b829a73cef9fd6deaef07035c9114078f17e7946bdf5e7cda3b17a98" exitCode=0
Sep 18 15:52:40.634912 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:40.634769    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b" event={"ID":"5c758c09-d7db-404e-a5cb-5c17ec90e7fb","Type":"ContainerDied","Data":"3aa5f518b829a73cef9fd6deaef07035c9114078f17e7946bdf5e7cda3b17a98"}
Sep 18 15:52:40.634912 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:40.634816    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b" event={"ID":"5c758c09-d7db-404e-a5cb-5c17ec90e7fb","Type":"ContainerStarted","Data":"1b4e2f667973f09d843d8ee7e6b23f755636298195ecf6ca71424deec36c75bc"}
Sep 18 15:52:40.672091 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:40.672066    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:41.390205 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:41.390169    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:41.456150 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:41.456119    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:41.561073 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:41.561040    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:41.788516 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:41.788491    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b"
Sep 18 15:52:41.858950 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:41.858919    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c758c09-d7db-404e-a5cb-5c17ec90e7fb-maas-api-service-ca\") pod \"5c758c09-d7db-404e-a5cb-5c17ec90e7fb\" (UID: \"5c758c09-d7db-404e-a5cb-5c17ec90e7fb\") "
Sep 18 15:52:41.859376 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:41.859349    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c758c09-d7db-404e-a5cb-5c17ec90e7fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c758c09-d7db-404e-a5cb-5c17ec90e7fb" (UID: "5c758c09-d7db-404e-a5cb-5c17ec90e7fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:52:41.925716 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:41.925684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:41.960714 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:41.960622    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c758c09-d7db-404e-a5cb-5c17ec90e7fb-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:42.024594 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.024562    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:42.123931 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.123894    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"]
Sep 18 15:52:42.124293 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.124272    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c758c09-d7db-404e-a5cb-5c17ec90e7fb" containerName="revoke-keys"
Sep 18 15:52:42.124293 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.124287    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c758c09-d7db-404e-a5cb-5c17ec90e7fb" containerName="revoke-keys"
Sep 18 15:52:42.124462 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.124361    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c758c09-d7db-404e-a5cb-5c17ec90e7fb" containerName="revoke-keys"
Sep 18 15:52:42.127518 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.127496    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"
Sep 18 15:52:42.130417 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.130397    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f84020ef\""
Sep 18 15:52:42.130849 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.130823    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f84020ef-dockercfg-ml9xh\""
Sep 18 15:52:42.145243 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.145220    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"]
Sep 18 15:52:42.162720 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.162684    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb3d6e7-bea3-43a6-b6e3-7737bd744d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s\" (UID: \"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"
Sep 18 15:52:42.162921 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.162771    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpldq\" (UniqueName: \"kubernetes.io/projected/cbb3d6e7-bea3-43a6-b6e3-7737bd744d79-kube-api-access-lpldq\") pod \"payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s\" (UID: \"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"
Sep 18 15:52:42.204946 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.204909    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"]
Sep 18 15:52:42.208332 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.208313    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"
Sep 18 15:52:42.222939 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.222912    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"]
Sep 18 15:52:42.263540 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.263504    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpldq\" (UniqueName: \"kubernetes.io/projected/cbb3d6e7-bea3-43a6-b6e3-7737bd744d79-kube-api-access-lpldq\") pod \"payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s\" (UID: \"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"
Sep 18 15:52:42.263709 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.263572    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxzb\" (UniqueName: \"kubernetes.io/projected/af036bce-d7c8-43a0-865b-08b6e02aa3bd-kube-api-access-dfxzb\") pod \"payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl\" (UID: \"af036bce-d7c8-43a0-865b-08b6e02aa3bd\") " pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"
Sep 18 15:52:42.263709 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.263651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af036bce-d7c8-43a0-865b-08b6e02aa3bd-plugins-config-volume\") pod \"payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl\" (UID: \"af036bce-d7c8-43a0-865b-08b6e02aa3bd\") " pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"
Sep 18 15:52:42.263805 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.263724    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb3d6e7-bea3-43a6-b6e3-7737bd744d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s\" (UID: \"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"
Sep 18 15:52:42.264641 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.264618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb3d6e7-bea3-43a6-b6e3-7737bd744d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s\" (UID: \"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"
Sep 18 15:52:42.273134 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.273113    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpldq\" (UniqueName: \"kubernetes.io/projected/cbb3d6e7-bea3-43a6-b6e3-7737bd744d79-kube-api-access-lpldq\") pod \"payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s\" (UID: \"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"
Sep 18 15:52:42.364785 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.364729    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxzb\" (UniqueName: \"kubernetes.io/projected/af036bce-d7c8-43a0-865b-08b6e02aa3bd-kube-api-access-dfxzb\") pod \"payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl\" (UID: \"af036bce-d7c8-43a0-865b-08b6e02aa3bd\") " pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"
Sep 18 15:52:42.364988 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.364808    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af036bce-d7c8-43a0-865b-08b6e02aa3bd-plugins-config-volume\") pod \"payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl\" (UID: \"af036bce-d7c8-43a0-865b-08b6e02aa3bd\") " pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"
Sep 18 15:52:42.365570 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.365549    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af036bce-d7c8-43a0-865b-08b6e02aa3bd-plugins-config-volume\") pod \"payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl\" (UID: \"af036bce-d7c8-43a0-865b-08b6e02aa3bd\") " pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"
Sep 18 15:52:42.374270 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.374242    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxzb\" (UniqueName: \"kubernetes.io/projected/af036bce-d7c8-43a0-865b-08b6e02aa3bd-kube-api-access-dfxzb\") pod \"payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl\" (UID: \"af036bce-d7c8-43a0-865b-08b6e02aa3bd\") " pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"
Sep 18 15:52:42.437778 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.437742    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"
Sep 18 15:52:42.494672 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.494606    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:42.518642 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.518605    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"
Sep 18 15:52:42.646338 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.646309    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b"
Sep 18 15:52:42.646532 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.646341    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b" event={"ID":"5c758c09-d7db-404e-a5cb-5c17ec90e7fb","Type":"ContainerDied","Data":"1b4e2f667973f09d843d8ee7e6b23f755636298195ecf6ca71424deec36c75bc"}
Sep 18 15:52:42.646532 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.646411    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b4e2f667973f09d843d8ee7e6b23f755636298195ecf6ca71424deec36c75bc"
Sep 18 15:52:42.713596 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.713570    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"]
Sep 18 15:52:42.715489 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:52:42.715463    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb3d6e7_bea3_43a6_b6e3_7737bd744d79.slice/crio-1db673bd2d697e85905a3950aef23acab14820fd82fac15fa70003c08a72fac0 WatchSource:0}: Error finding container 1db673bd2d697e85905a3950aef23acab14820fd82fac15fa70003c08a72fac0: Status 404 returned error can't find the container with id 1db673bd2d697e85905a3950aef23acab14820fd82fac15fa70003c08a72fac0
Sep 18 15:52:42.803469 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:42.803440    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"]
Sep 18 15:52:42.805539 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:52:42.805501    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf036bce_d7c8_43a0_865b_08b6e02aa3bd.slice/crio-35fd81e85126853e425f78b1e4497fc4d8b6904c3ae3a91479904a7fcc3702b7 WatchSource:0}: Error finding container 35fd81e85126853e425f78b1e4497fc4d8b6904c3ae3a91479904a7fcc3702b7: Status 404 returned error can't find the container with id 35fd81e85126853e425f78b1e4497fc4d8b6904c3ae3a91479904a7fcc3702b7
Sep 18 15:52:43.381429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:43.381389    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:43.653269 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:43.653164    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s" event={"ID":"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79","Type":"ContainerStarted","Data":"85684586ebb7415eb46e2eb50288a62989e0beecbd5775ecc032b5cf31969c86"}
Sep 18 15:52:43.653445 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:43.653328    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"
Sep 18 15:52:43.653445 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:43.653366    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s" event={"ID":"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79","Type":"ContainerStarted","Data":"1db673bd2d697e85905a3950aef23acab14820fd82fac15fa70003c08a72fac0"}
Sep 18 15:52:43.654787 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:43.654760    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl" event={"ID":"af036bce-d7c8-43a0-865b-08b6e02aa3bd","Type":"ContainerStarted","Data":"932d5708f3b287fa0e201f5ad74de00a0c82d4cc364c3bd3dc652451e890b83e"}
Sep 18 15:52:43.654927 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:43.654808    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl" event={"ID":"af036bce-d7c8-43a0-865b-08b6e02aa3bd","Type":"ContainerStarted","Data":"35fd81e85126853e425f78b1e4497fc4d8b6904c3ae3a91479904a7fcc3702b7"}
Sep 18 15:52:43.654927 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:43.654885    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"
Sep 18 15:52:43.672220 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:43.672169    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s" podStartSLOduration=1.672153909 podStartE2EDuration="1.672153909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:52:43.671063299 +0000 UTC m=+1086.232295473" watchObservedRunningTime="2026-09-18 15:52:43.672153909 +0000 UTC m=+1086.233386048"
Sep 18 15:52:43.696024 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:43.695969    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl" podStartSLOduration=1.695953019 podStartE2EDuration="1.695953019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:52:43.694680411 +0000 UTC m=+1086.255912552" watchObservedRunningTime="2026-09-18 15:52:43.695953019 +0000 UTC m=+1086.257185159"
Sep 18 15:52:44.230646 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:44.230596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:48.210674 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:48.210640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:49.265483 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:49.265438    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:49.965706 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:49.965661    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:50.006434 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.006398    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"]
Sep 18 15:52:50.006679 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.006644    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2" podUID="b94b76d4-ea0f-486d-b9ca-f1375e72bcf4" containerName="payload-pre-processing" containerID="cri-o://9b4406676dc5ecffa2873b3e28636928c84470f1e94c863f38091c7edacda5c9" gracePeriod=30
Sep 18 15:52:50.066162 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.066123    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:50.166384 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.166349    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:50.197451 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.197421    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b"]
Sep 18 15:52:50.201607 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.201577    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-70c6df-xh45b"]
Sep 18 15:52:50.260468 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.260442    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"
Sep 18 15:52:50.266540 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.266508    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:50.337745 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.337712    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94b76d4-ea0f-486d-b9ca-f1375e72bcf4-plugins-config-volume\") pod \"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4\" (UID: \"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4\") "
Sep 18 15:52:50.337975 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.337798    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj79z\" (UniqueName: \"kubernetes.io/projected/b94b76d4-ea0f-486d-b9ca-f1375e72bcf4-kube-api-access-lj79z\") pod \"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4\" (UID: \"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4\") "
Sep 18 15:52:50.338099 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.338077    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94b76d4-ea0f-486d-b9ca-f1375e72bcf4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b94b76d4-ea0f-486d-b9ca-f1375e72bcf4" (UID: "b94b76d4-ea0f-486d-b9ca-f1375e72bcf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:52:50.338193 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.338174    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94b76d4-ea0f-486d-b9ca-f1375e72bcf4-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:50.339836 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.339809    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94b76d4-ea0f-486d-b9ca-f1375e72bcf4-kube-api-access-lj79z" (OuterVolumeSpecName: "kube-api-access-lj79z") pod "b94b76d4-ea0f-486d-b9ca-f1375e72bcf4" (UID: "b94b76d4-ea0f-486d-b9ca-f1375e72bcf4"). InnerVolumeSpecName "kube-api-access-lj79z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:52:50.438822 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.438786    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj79z\" (UniqueName: \"kubernetes.io/projected/b94b76d4-ea0f-486d-b9ca-f1375e72bcf4-kube-api-access-lj79z\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:52:50.554573 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.554480    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:50.653878 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.653814    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:50.692180 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.692148    2559 generic.go:356] "Generic (PLEG): container finished" podID="b94b76d4-ea0f-486d-b9ca-f1375e72bcf4" containerID="9b4406676dc5ecffa2873b3e28636928c84470f1e94c863f38091c7edacda5c9" exitCode=0
Sep 18 15:52:50.692343 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.692188    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"
Sep 18 15:52:50.692343 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.692230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2" event={"ID":"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4","Type":"ContainerDied","Data":"9b4406676dc5ecffa2873b3e28636928c84470f1e94c863f38091c7edacda5c9"}
Sep 18 15:52:50.692343 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.692287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2" event={"ID":"b94b76d4-ea0f-486d-b9ca-f1375e72bcf4","Type":"ContainerDied","Data":"308bef90befd308e19bc46f46573fd3403d777d4fd5bed45ac56639c3d149356"}
Sep 18 15:52:50.692343 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.692310    2559 scope.go:117] "RemoveContainer" containerID="9b4406676dc5ecffa2873b3e28636928c84470f1e94c863f38091c7edacda5c9"
Sep 18 15:52:50.701427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.701401    2559 scope.go:117] "RemoveContainer" containerID="9b4406676dc5ecffa2873b3e28636928c84470f1e94c863f38091c7edacda5c9"
Sep 18 15:52:50.701836 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:52:50.701769    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b4406676dc5ecffa2873b3e28636928c84470f1e94c863f38091c7edacda5c9\": container with ID starting with 9b4406676dc5ecffa2873b3e28636928c84470f1e94c863f38091c7edacda5c9 not found: ID does not exist" containerID="9b4406676dc5ecffa2873b3e28636928c84470f1e94c863f38091c7edacda5c9"
Sep 18 15:52:50.701973 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.701893    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b4406676dc5ecffa2873b3e28636928c84470f1e94c863f38091c7edacda5c9"} err="failed to get container status \"9b4406676dc5ecffa2873b3e28636928c84470f1e94c863f38091c7edacda5c9\": rpc error: code = NotFound desc = could not find container \"9b4406676dc5ecffa2873b3e28636928c84470f1e94c863f38091c7edacda5c9\": container with ID starting with 9b4406676dc5ecffa2873b3e28636928c84470f1e94c863f38091c7edacda5c9 not found: ID does not exist"
Sep 18 15:52:50.719032 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.719007    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"]
Sep 18 15:52:50.722117 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:50.722095    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70c6df-77b4b5656b-cl5f2"]
Sep 18 15:52:51.386729 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:51.386695    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:51.744128 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:51.744094    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c758c09-d7db-404e-a5cb-5c17ec90e7fb" path="/var/lib/kubelet/pods/5c758c09-d7db-404e-a5cb-5c17ec90e7fb/volumes"
Sep 18 15:52:51.744469 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:51.744457    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b94b76d4-ea0f-486d-b9ca-f1375e72bcf4" path="/var/lib/kubelet/pods/b94b76d4-ea0f-486d-b9ca-f1375e72bcf4/volumes"
Sep 18 15:52:52.181393 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:52.181362    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:52.791818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:52.791784    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:54.401560 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:54.401526    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:54.663281 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:54.663197    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"
Sep 18 15:52:54.663281 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:54.663270    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"
Sep 18 15:52:55.384224 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:55.384189    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:56.206354 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:56.206317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:56.812865 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:56.812814    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:56.911429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:56.911390    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:57.103259 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:57.103152    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:57.198739 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:57.198694    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:57.395705 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:57.395611    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:52:57.665123 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:57.665026    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:57.947222 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:57.947189    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:58.137328 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:58.137296    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:58.324256 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:58.324100    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:58.980439 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:58.980404    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:59.076541 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:59.076502    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:52:59.382148 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:52:59.382050    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:00.634540 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:00.634502    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:01.255508 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:01.255475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:01.360686 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:01.360643    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:01.475845 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:01.475805    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:02.424145 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:02.424111    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:03.601475 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:03.601436    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:04.586105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:04.586065    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:04.693233 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:04.693194    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:04.898143 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:04.898056    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:05.108183 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.108146    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:05.203035 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.202978    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:05.285914 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.285874    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"]
Sep 18 15:53:05.286269 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.286256    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94b76d4-ea0f-486d-b9ca-f1375e72bcf4" containerName="payload-pre-processing"
Sep 18 15:53:05.286328 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.286271    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94b76d4-ea0f-486d-b9ca-f1375e72bcf4" containerName="payload-pre-processing"
Sep 18 15:53:05.286368 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.286350    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94b76d4-ea0f-486d-b9ca-f1375e72bcf4" containerName="payload-pre-processing"
Sep 18 15:53:05.289485 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.289464    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"
Sep 18 15:53:05.293081 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.293058    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-f14aba\""
Sep 18 15:53:05.293200 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.293149    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-f14aba-dockercfg-ps7wk\""
Sep 18 15:53:05.301175 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.301149    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"]
Sep 18 15:53:05.353422 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.353393    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"]
Sep 18 15:53:05.357105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.357085    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"
Sep 18 15:53:05.369316 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.369275    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"]
Sep 18 15:53:05.392058 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.392028    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2vn4\" (UniqueName: \"kubernetes.io/projected/b261b9f5-63ab-459e-894d-d8d9898c95f0-kube-api-access-b2vn4\") pod \"payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft\" (UID: \"b261b9f5-63ab-459e-894d-d8d9898c95f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"
Sep 18 15:53:05.392256 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.392139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b261b9f5-63ab-459e-894d-d8d9898c95f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft\" (UID: \"b261b9f5-63ab-459e-894d-d8d9898c95f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"
Sep 18 15:53:05.493446 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.493361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b261b9f5-63ab-459e-894d-d8d9898c95f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft\" (UID: \"b261b9f5-63ab-459e-894d-d8d9898c95f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"
Sep 18 15:53:05.493446 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.493426    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjmg\" (UniqueName: \"kubernetes.io/projected/82afb053-d882-4da8-8544-bea819831570-kube-api-access-mpjmg\") pod \"payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb\" (UID: \"82afb053-d882-4da8-8544-bea819831570\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"
Sep 18 15:53:05.493651 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.493500    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82afb053-d882-4da8-8544-bea819831570-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb\" (UID: \"82afb053-d882-4da8-8544-bea819831570\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"
Sep 18 15:53:05.493651 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.493576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2vn4\" (UniqueName: \"kubernetes.io/projected/b261b9f5-63ab-459e-894d-d8d9898c95f0-kube-api-access-b2vn4\") pod \"payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft\" (UID: \"b261b9f5-63ab-459e-894d-d8d9898c95f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"
Sep 18 15:53:05.494105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.494084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b261b9f5-63ab-459e-894d-d8d9898c95f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft\" (UID: \"b261b9f5-63ab-459e-894d-d8d9898c95f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"
Sep 18 15:53:05.502205 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.502177    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2vn4\" (UniqueName: \"kubernetes.io/projected/b261b9f5-63ab-459e-894d-d8d9898c95f0-kube-api-access-b2vn4\") pod \"payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft\" (UID: \"b261b9f5-63ab-459e-894d-d8d9898c95f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"
Sep 18 15:53:05.595073 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.595037    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjmg\" (UniqueName: \"kubernetes.io/projected/82afb053-d882-4da8-8544-bea819831570-kube-api-access-mpjmg\") pod \"payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb\" (UID: \"82afb053-d882-4da8-8544-bea819831570\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"
Sep 18 15:53:05.595234 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.595116    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82afb053-d882-4da8-8544-bea819831570-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb\" (UID: \"82afb053-d882-4da8-8544-bea819831570\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"
Sep 18 15:53:05.596103 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.596078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82afb053-d882-4da8-8544-bea819831570-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb\" (UID: \"82afb053-d882-4da8-8544-bea819831570\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"
Sep 18 15:53:05.600868 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.600839    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"
Sep 18 15:53:05.604912 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.604888    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjmg\" (UniqueName: \"kubernetes.io/projected/82afb053-d882-4da8-8544-bea819831570-kube-api-access-mpjmg\") pod \"payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb\" (UID: \"82afb053-d882-4da8-8544-bea819831570\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"
Sep 18 15:53:05.610275 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.610243    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:05.673298 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.673264    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"
Sep 18 15:53:05.874906 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.874876    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"]
Sep 18 15:53:05.875508 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:53:05.875479    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb261b9f5_63ab_459e_894d_d8d9898c95f0.slice/crio-73909df8d81ef77b3df3b003723e8d1764aaa7265edce5be683ffa9b497162e8 WatchSource:0}: Error finding container 73909df8d81ef77b3df3b003723e8d1764aaa7265edce5be683ffa9b497162e8: Status 404 returned error can't find the container with id 73909df8d81ef77b3df3b003723e8d1764aaa7265edce5be683ffa9b497162e8
Sep 18 15:53:05.958489 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:05.958461    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"]
Sep 18 15:53:05.960508 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:53:05.960469    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82afb053_d882_4da8_8544_bea819831570.slice/crio-fb97185dd7cbc6ba4f9008975a631c641bd98fb7607721bcec867d26bfaf2bdf WatchSource:0}: Error finding container fb97185dd7cbc6ba4f9008975a631c641bd98fb7607721bcec867d26bfaf2bdf: Status 404 returned error can't find the container with id fb97185dd7cbc6ba4f9008975a631c641bd98fb7607721bcec867d26bfaf2bdf
Sep 18 15:53:06.368189 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:06.368145    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:06.698382 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:06.698340    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:06.790836 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:06.790796    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb" event={"ID":"82afb053-d882-4da8-8544-bea819831570","Type":"ContainerStarted","Data":"f467da982fa651a91ee460a1b6f5db9d0b5d9298a32210ddae53df8f9a8f587b"}
Sep 18 15:53:06.791038 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:06.790871    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb" event={"ID":"82afb053-d882-4da8-8544-bea819831570","Type":"ContainerStarted","Data":"fb97185dd7cbc6ba4f9008975a631c641bd98fb7607721bcec867d26bfaf2bdf"}
Sep 18 15:53:06.791038 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:06.790902    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"
Sep 18 15:53:06.793022 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:06.792994    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft" event={"ID":"b261b9f5-63ab-459e-894d-d8d9898c95f0","Type":"ContainerStarted","Data":"4d6763cd444290cd4076bfcc6f5d4159a00eb3acd12bbf044b78b0b636e54e32"}
Sep 18 15:53:06.793122 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:06.793048    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft" event={"ID":"b261b9f5-63ab-459e-894d-d8d9898c95f0","Type":"ContainerStarted","Data":"73909df8d81ef77b3df3b003723e8d1764aaa7265edce5be683ffa9b497162e8"}
Sep 18 15:53:06.793122 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:06.793077    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"
Sep 18 15:53:06.798238 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:06.798213    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:06.812534 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:06.812481    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb" podStartSLOduration=1.812461409 podStartE2EDuration="1.812461409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:53:06.810159481 +0000 UTC m=+1109.371391622" watchObservedRunningTime="2026-09-18 15:53:06.812461409 +0000 UTC m=+1109.373693552"
Sep 18 15:53:06.828330 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:06.828289    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft" podStartSLOduration=1.828277446 podStartE2EDuration="1.828277446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:53:06.827046761 +0000 UTC m=+1109.388278901" watchObservedRunningTime="2026-09-18 15:53:06.828277446 +0000 UTC m=+1109.389509587"
Sep 18 15:53:07.048329 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:07.048249    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl"]
Sep 18 15:53:07.052602 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:07.052561    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl"
Sep 18 15:53:07.055830 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:07.055591    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 15:53:07.055830 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:07.055595    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 15:53:07.059385 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:07.059321    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl"]
Sep 18 15:53:07.221362 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:07.221319    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c0509e-2629-455a-b410-acb7219955ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl\" (UID: \"a9c0509e-2629-455a-b410-acb7219955ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl"
Sep 18 15:53:07.322126 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:07.322033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c0509e-2629-455a-b410-acb7219955ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl\" (UID: \"a9c0509e-2629-455a-b410-acb7219955ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl"
Sep 18 15:53:07.322658 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:07.322639    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c0509e-2629-455a-b410-acb7219955ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl\" (UID: \"a9c0509e-2629-455a-b410-acb7219955ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl"
Sep 18 15:53:07.368966 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:07.368932    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl"
Sep 18 15:53:07.476044 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:07.476003    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:07.643426 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:07.643392    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl"]
Sep 18 15:53:07.645028 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:53:07.644998    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c0509e_2629_455a_b410_acb7219955ca.slice/crio-f6830cefb9f19ea964f29e2d624af12f69ae9ce327526d2eb50122cb8baaadf7 WatchSource:0}: Error finding container f6830cefb9f19ea964f29e2d624af12f69ae9ce327526d2eb50122cb8baaadf7: Status 404 returned error can't find the container with id f6830cefb9f19ea964f29e2d624af12f69ae9ce327526d2eb50122cb8baaadf7
Sep 18 15:53:07.800245 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:07.800149    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl" event={"ID":"a9c0509e-2629-455a-b410-acb7219955ca","Type":"ContainerStarted","Data":"f6830cefb9f19ea964f29e2d624af12f69ae9ce327526d2eb50122cb8baaadf7"}
Sep 18 15:53:08.587621 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:08.587587    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:08.623727 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:08.623691    2559 patch_prober.go:28] interesting pod/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 18 15:53:08.623915 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:08.623749    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 18 15:53:08.807119 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:08.807081    2559 generic.go:356] "Generic (PLEG): container finished" podID="a9c0509e-2629-455a-b410-acb7219955ca" containerID="fe0ee7e3a9aafb7f17e2d2a39966475f480a1c393a14d94df113c53b122fa02a" exitCode=0
Sep 18 15:53:08.807294 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:08.807149    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl" event={"ID":"a9c0509e-2629-455a-b410-acb7219955ca","Type":"ContainerDied","Data":"fe0ee7e3a9aafb7f17e2d2a39966475f480a1c393a14d94df113c53b122fa02a"}
Sep 18 15:53:08.808907 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:08.808887    2559 generic.go:356] "Generic (PLEG): container finished" podID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerID="f2966897a5b54e63f77b765fb9152e2013f75e7c4e307d8335578004740dd816" exitCode=0
Sep 18 15:53:08.809019 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:08.808944    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" event={"ID":"b1bee9c4-37ea-4909-8e44-af4bd98fb252","Type":"ContainerDied","Data":"f2966897a5b54e63f77b765fb9152e2013f75e7c4e307d8335578004740dd816"}
Sep 18 15:53:08.809309 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:08.809296    2559 scope.go:117] "RemoveContainer" containerID="f2966897a5b54e63f77b765fb9152e2013f75e7c4e307d8335578004740dd816"
Sep 18 15:53:08.810396 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:08.810367    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:53:08.810464 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:08.810435    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:53:08.810503 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:08.810467    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:53:09.464662 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:09.464616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:09.816890 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:09.816796    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" event={"ID":"b1bee9c4-37ea-4909-8e44-af4bd98fb252","Type":"ContainerStarted","Data":"dba98aad75549ccf74282e42d9c3182a449e0c37aa7f8fce7d355447ac60199f"}
Sep 18 15:53:09.941565 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:09.941528    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:53:09.941789 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:09.941765    2559 patch_prober.go:28] interesting pod/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 18 15:53:09.941848 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:09.941824    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 18 15:53:10.000537 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:10.000511    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl"
Sep 18 15:53:10.048472 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:10.048438    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:10.048654 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:10.048496    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c0509e-2629-455a-b410-acb7219955ca-maas-api-service-ca\") pod \"a9c0509e-2629-455a-b410-acb7219955ca\" (UID: \"a9c0509e-2629-455a-b410-acb7219955ca\") "
Sep 18 15:53:10.048941 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:10.048919    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c0509e-2629-455a-b410-acb7219955ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9c0509e-2629-455a-b410-acb7219955ca" (UID: "a9c0509e-2629-455a-b410-acb7219955ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:53:10.149415 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:10.149331    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9c0509e-2629-455a-b410-acb7219955ca-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:10.822851 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:10.822818    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl"
Sep 18 15:53:10.823292 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:10.822816    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl" event={"ID":"a9c0509e-2629-455a-b410-acb7219955ca","Type":"ContainerDied","Data":"f6830cefb9f19ea964f29e2d624af12f69ae9ce327526d2eb50122cb8baaadf7"}
Sep 18 15:53:10.823292 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:10.822968    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6830cefb9f19ea964f29e2d624af12f69ae9ce327526d2eb50122cb8baaadf7"
Sep 18 15:53:10.887476 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:10.887441    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:10.941695 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:10.941655    2559 patch_prober.go:28] interesting pod/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 18 15:53:10.941910 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:10.941715    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 18 15:53:11.688569 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:11.688527    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:11.941840 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:11.941735    2559 patch_prober.go:28] interesting pod/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 18 15:53:11.942279 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:11.941841    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 18 15:53:12.941952 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:12.941917    2559 patch_prober.go:28] interesting pod/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 18 15:53:12.942415 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:12.941974    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 18 15:53:13.942617 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:13.942580    2559 patch_prober.go:28] interesting pod/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 18 15:53:13.943085 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:13.942676    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 18 15:53:14.948730 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:14.948704    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:53:14.949224 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:14.949164    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:53:14.951488 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:14.951465    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:53:17.802624 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:17.802594    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"
Sep 18 15:53:17.803017 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:17.802663    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"
Sep 18 15:53:20.369211 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.369173    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:20.397269 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.397237    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"]
Sep 18 15:53:20.397500 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.397450    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl" podUID="af036bce-d7c8-43a0-865b-08b6e02aa3bd" containerName="payload-processing" containerID="cri-o://932d5708f3b287fa0e201f5ad74de00a0c82d4cc364c3bd3dc652451e890b83e" gracePeriod=30
Sep 18 15:53:20.405526 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.405166    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"]
Sep 18 15:53:20.405526 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.405373    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s" podUID="cbb3d6e7-bea3-43a6-b6e3-7737bd744d79" containerName="payload-pre-processing" containerID="cri-o://85684586ebb7415eb46e2eb50288a62989e0beecbd5775ecc032b5cf31969c86" gracePeriod=30
Sep 18 15:53:20.441649 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.441618    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:20.511058 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.511019    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:20.585672 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.585644    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl"]
Sep 18 15:53:20.589319 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.589290    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f84020ef-d8bkl"]
Sep 18 15:53:20.590145 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.590112    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:20.689526 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.689495    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"
Sep 18 15:53:20.692468 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.692446    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"
Sep 18 15:53:20.761019 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.760982    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfxzb\" (UniqueName: \"kubernetes.io/projected/af036bce-d7c8-43a0-865b-08b6e02aa3bd-kube-api-access-dfxzb\") pod \"af036bce-d7c8-43a0-865b-08b6e02aa3bd\" (UID: \"af036bce-d7c8-43a0-865b-08b6e02aa3bd\") "
Sep 18 15:53:20.761219 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.761041    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af036bce-d7c8-43a0-865b-08b6e02aa3bd-plugins-config-volume\") pod \"af036bce-d7c8-43a0-865b-08b6e02aa3bd\" (UID: \"af036bce-d7c8-43a0-865b-08b6e02aa3bd\") "
Sep 18 15:53:20.761219 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.761097    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpldq\" (UniqueName: \"kubernetes.io/projected/cbb3d6e7-bea3-43a6-b6e3-7737bd744d79-kube-api-access-lpldq\") pod \"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79\" (UID: \"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79\") "
Sep 18 15:53:20.761219 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.761204    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb3d6e7-bea3-43a6-b6e3-7737bd744d79-plugins-config-volume\") pod \"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79\" (UID: \"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79\") "
Sep 18 15:53:20.762064 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.761986    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb3d6e7-bea3-43a6-b6e3-7737bd744d79-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cbb3d6e7-bea3-43a6-b6e3-7737bd744d79" (UID: "cbb3d6e7-bea3-43a6-b6e3-7737bd744d79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:53:20.762418 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.762389    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af036bce-d7c8-43a0-865b-08b6e02aa3bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af036bce-d7c8-43a0-865b-08b6e02aa3bd" (UID: "af036bce-d7c8-43a0-865b-08b6e02aa3bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:53:20.763635 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.763600    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af036bce-d7c8-43a0-865b-08b6e02aa3bd-kube-api-access-dfxzb" (OuterVolumeSpecName: "kube-api-access-dfxzb") pod "af036bce-d7c8-43a0-865b-08b6e02aa3bd" (UID: "af036bce-d7c8-43a0-865b-08b6e02aa3bd"). InnerVolumeSpecName "kube-api-access-dfxzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:53:20.764504 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.764481    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb3d6e7-bea3-43a6-b6e3-7737bd744d79-kube-api-access-lpldq" (OuterVolumeSpecName: "kube-api-access-lpldq") pod "cbb3d6e7-bea3-43a6-b6e3-7737bd744d79" (UID: "cbb3d6e7-bea3-43a6-b6e3-7737bd744d79"). InnerVolumeSpecName "kube-api-access-lpldq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:53:20.863028 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.862994    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfxzb\" (UniqueName: \"kubernetes.io/projected/af036bce-d7c8-43a0-865b-08b6e02aa3bd-kube-api-access-dfxzb\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:20.863028 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.863024    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af036bce-d7c8-43a0-865b-08b6e02aa3bd-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:20.863028 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.863034    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpldq\" (UniqueName: \"kubernetes.io/projected/cbb3d6e7-bea3-43a6-b6e3-7737bd744d79-kube-api-access-lpldq\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:20.863251 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.863043    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb3d6e7-bea3-43a6-b6e3-7737bd744d79-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:20.883630 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.883600    2559 generic.go:356] "Generic (PLEG): container finished" podID="af036bce-d7c8-43a0-865b-08b6e02aa3bd" containerID="932d5708f3b287fa0e201f5ad74de00a0c82d4cc364c3bd3dc652451e890b83e" exitCode=0
Sep 18 15:53:20.883754 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.883659    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"
Sep 18 15:53:20.883754 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.883697    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl" event={"ID":"af036bce-d7c8-43a0-865b-08b6e02aa3bd","Type":"ContainerDied","Data":"932d5708f3b287fa0e201f5ad74de00a0c82d4cc364c3bd3dc652451e890b83e"}
Sep 18 15:53:20.883839 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.883784    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl" event={"ID":"af036bce-d7c8-43a0-865b-08b6e02aa3bd","Type":"ContainerDied","Data":"35fd81e85126853e425f78b1e4497fc4d8b6904c3ae3a91479904a7fcc3702b7"}
Sep 18 15:53:20.883839 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.883817    2559 scope.go:117] "RemoveContainer" containerID="932d5708f3b287fa0e201f5ad74de00a0c82d4cc364c3bd3dc652451e890b83e"
Sep 18 15:53:20.884927 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.884908    2559 generic.go:356] "Generic (PLEG): container finished" podID="cbb3d6e7-bea3-43a6-b6e3-7737bd744d79" containerID="85684586ebb7415eb46e2eb50288a62989e0beecbd5775ecc032b5cf31969c86" exitCode=0
Sep 18 15:53:20.884989 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.884955    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"
Sep 18 15:53:20.885029 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.884983    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s" event={"ID":"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79","Type":"ContainerDied","Data":"85684586ebb7415eb46e2eb50288a62989e0beecbd5775ecc032b5cf31969c86"}
Sep 18 15:53:20.885074 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.885032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s" event={"ID":"cbb3d6e7-bea3-43a6-b6e3-7737bd744d79","Type":"ContainerDied","Data":"1db673bd2d697e85905a3950aef23acab14820fd82fac15fa70003c08a72fac0"}
Sep 18 15:53:20.892716 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.892692    2559 scope.go:117] "RemoveContainer" containerID="932d5708f3b287fa0e201f5ad74de00a0c82d4cc364c3bd3dc652451e890b83e"
Sep 18 15:53:20.893071 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:53:20.893033    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932d5708f3b287fa0e201f5ad74de00a0c82d4cc364c3bd3dc652451e890b83e\": container with ID starting with 932d5708f3b287fa0e201f5ad74de00a0c82d4cc364c3bd3dc652451e890b83e not found: ID does not exist" containerID="932d5708f3b287fa0e201f5ad74de00a0c82d4cc364c3bd3dc652451e890b83e"
Sep 18 15:53:20.893142 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.893070    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932d5708f3b287fa0e201f5ad74de00a0c82d4cc364c3bd3dc652451e890b83e"} err="failed to get container status \"932d5708f3b287fa0e201f5ad74de00a0c82d4cc364c3bd3dc652451e890b83e\": rpc error: code = NotFound desc = could not find container \"932d5708f3b287fa0e201f5ad74de00a0c82d4cc364c3bd3dc652451e890b83e\": container with ID starting with 932d5708f3b287fa0e201f5ad74de00a0c82d4cc364c3bd3dc652451e890b83e not found: ID does not exist"
Sep 18 15:53:20.893142 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.893108    2559 scope.go:117] "RemoveContainer" containerID="85684586ebb7415eb46e2eb50288a62989e0beecbd5775ecc032b5cf31969c86"
Sep 18 15:53:20.901919 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.901903    2559 scope.go:117] "RemoveContainer" containerID="85684586ebb7415eb46e2eb50288a62989e0beecbd5775ecc032b5cf31969c86"
Sep 18 15:53:20.902180 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:53:20.902162    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85684586ebb7415eb46e2eb50288a62989e0beecbd5775ecc032b5cf31969c86\": container with ID starting with 85684586ebb7415eb46e2eb50288a62989e0beecbd5775ecc032b5cf31969c86 not found: ID does not exist" containerID="85684586ebb7415eb46e2eb50288a62989e0beecbd5775ecc032b5cf31969c86"
Sep 18 15:53:20.902233 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.902186    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85684586ebb7415eb46e2eb50288a62989e0beecbd5775ecc032b5cf31969c86"} err="failed to get container status \"85684586ebb7415eb46e2eb50288a62989e0beecbd5775ecc032b5cf31969c86\": rpc error: code = NotFound desc = could not find container \"85684586ebb7415eb46e2eb50288a62989e0beecbd5775ecc032b5cf31969c86\": container with ID starting with 85684586ebb7415eb46e2eb50288a62989e0beecbd5775ecc032b5cf31969c86 not found: ID does not exist"
Sep 18 15:53:20.908678 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.908628    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"]
Sep 18 15:53:20.914728 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.914703    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f84020ef-6b788569b6-qzx4s"]
Sep 18 15:53:20.925876 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.925844    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"]
Sep 18 15:53:20.929830 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.929798    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f84020ef-599f9b8977-skmvl"]
Sep 18 15:53:20.942463 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:20.942425    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:21.043174 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:21.043138    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:21.705244 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:21.705193    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:21.744415 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:21.744382    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9c0509e-2629-455a-b410-acb7219955ca" path="/var/lib/kubelet/pods/a9c0509e-2629-455a-b410-acb7219955ca/volumes"
Sep 18 15:53:21.744752 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:21.744738    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af036bce-d7c8-43a0-865b-08b6e02aa3bd" path="/var/lib/kubelet/pods/af036bce-d7c8-43a0-865b-08b6e02aa3bd/volumes"
Sep 18 15:53:21.745097 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:21.745084    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbb3d6e7-bea3-43a6-b6e3-7737bd744d79" path="/var/lib/kubelet/pods/cbb3d6e7-bea3-43a6-b6e3-7737bd744d79/volumes"
Sep 18 15:53:22.596614 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:22.596581    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:23.677414 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:23.677378    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:24.289647 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:24.289616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:25.122351 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:25.122322    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:26.022323 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.022291    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:26.143079 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.143034    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:26.479822 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.479789    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:26.750720 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.750625    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"]
Sep 18 15:53:26.751161 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.751142    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af036bce-d7c8-43a0-865b-08b6e02aa3bd" containerName="payload-processing"
Sep 18 15:53:26.751267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.751164    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="af036bce-d7c8-43a0-865b-08b6e02aa3bd" containerName="payload-processing"
Sep 18 15:53:26.751267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.751178    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9c0509e-2629-455a-b410-acb7219955ca" containerName="revoke-keys"
Sep 18 15:53:26.751267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.751185    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9c0509e-2629-455a-b410-acb7219955ca" containerName="revoke-keys"
Sep 18 15:53:26.751267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.751199    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb3d6e7-bea3-43a6-b6e3-7737bd744d79" containerName="payload-pre-processing"
Sep 18 15:53:26.751267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.751207    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb3d6e7-bea3-43a6-b6e3-7737bd744d79" containerName="payload-pre-processing"
Sep 18 15:53:26.751513 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.751294    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9c0509e-2629-455a-b410-acb7219955ca" containerName="revoke-keys"
Sep 18 15:53:26.751513 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.751307    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="af036bce-d7c8-43a0-865b-08b6e02aa3bd" containerName="payload-processing"
Sep 18 15:53:26.751513 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.751320    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb3d6e7-bea3-43a6-b6e3-7737bd744d79" containerName="payload-pre-processing"
Sep 18 15:53:26.753660 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.753638    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.756971 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.756947    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4e81aa-openshift-default-dockercfg-cthkq\""
Sep 18 15:53:26.767651 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.767624    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"]
Sep 18 15:53:26.780759 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.780730    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:26.822695 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.822663    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-workload-certs\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.822881 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.822704    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istiod-ca-cert\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.822881 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.822728    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-podinfo\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.822881 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.822816    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-workload-socket\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.822881 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.822841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-credential-socket\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.823036 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.822888    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-envoy\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.823036 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.822940    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp5jw\" (UniqueName: \"kubernetes.io/projected/af9fcf83-d452-4650-bbcd-a5d13a7e8884-kube-api-access-sp5jw\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.823036 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.822975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-token\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.823036 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.823004    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-data\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.910516 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.910484    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:26.923769 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.923737    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-data\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.923954 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.923790    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-workload-certs\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.923954 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.923821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istiod-ca-cert\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.923954 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.923845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-podinfo\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.923954 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.923887    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-workload-socket\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.923954 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.923907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-credential-socket\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.924201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.924034    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-envoy\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.924201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.924100    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp5jw\" (UniqueName: \"kubernetes.io/projected/af9fcf83-d452-4650-bbcd-a5d13a7e8884-kube-api-access-sp5jw\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.924201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.924157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-token\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.924201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.924178    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-data\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.924413 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.924302    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-workload-socket\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.924413 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.924322    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-credential-socket\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.924611 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.924592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istiod-ca-cert\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.924683 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.924630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-workload-certs\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.926357 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.926325    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-envoy\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.926530 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.926513    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-podinfo\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.934830 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.934802    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp5jw\" (UniqueName: \"kubernetes.io/projected/af9fcf83-d452-4650-bbcd-a5d13a7e8884-kube-api-access-sp5jw\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:26.936170 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:26.936152    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-token\") pod \"e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") " pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:27.064485 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:27.064382    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:27.114297 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:27.114211    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:27.344554 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:27.344527    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"]
Sep 18 15:53:27.346169 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:53:27.346143    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9fcf83_d452_4650_bbcd_a5d13a7e8884.slice/crio-7848257d75b9fc36ca31f5b476c4231ecc753841b31a7ffe6d518fd933e4fdd7 WatchSource:0}: Error finding container 7848257d75b9fc36ca31f5b476c4231ecc753841b31a7ffe6d518fd933e4fdd7: Status 404 returned error can't find the container with id 7848257d75b9fc36ca31f5b476c4231ecc753841b31a7ffe6d518fd933e4fdd7
Sep 18 15:53:27.348572 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:27.348539    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:53:27.348649 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:27.348610    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:53:27.348649 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:27.348638    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:53:27.827269 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:27.827227    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:27.924954 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:27.924918    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" event={"ID":"af9fcf83-d452-4650-bbcd-a5d13a7e8884","Type":"ContainerStarted","Data":"66398535e44629b3fb3c976c20ade87e2d358d896e67d1587b556c3cd4d51053"}
Sep 18 15:53:27.925126 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:27.924980    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" event={"ID":"af9fcf83-d452-4650-bbcd-a5d13a7e8884","Type":"ContainerStarted","Data":"7848257d75b9fc36ca31f5b476c4231ecc753841b31a7ffe6d518fd933e4fdd7"}
Sep 18 15:53:27.958222 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:27.958168    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" podStartSLOduration=1.958150136 podStartE2EDuration="1.958150136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:53:27.956334237 +0000 UTC m=+1130.517566402" watchObservedRunningTime="2026-09-18 15:53:27.958150136 +0000 UTC m=+1130.519382346"
Sep 18 15:53:28.065234 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:28.065197    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:28.066918 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:28.066893    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 18 15:53:28.067010 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:28.066947    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 18 15:53:29.070135 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:29.070100    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:29.310943 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:29.310904    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:29.406178 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:29.406058    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:29.525216 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:29.525180    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:29.569737 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:29.569704    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"]
Sep 18 15:53:29.569944 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:29.569919    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy" containerID="cri-o://dba98aad75549ccf74282e42d9c3182a449e0c37aa7f8fce7d355447ac60199f" gracePeriod=30
Sep 18 15:53:29.626887 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:29.626823    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:29.913436 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:29.913388    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:29.944873 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:29.944818    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:29.946162 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:29.946132    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:53:29.951573 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:29.951542    2559 patch_prober.go:28] interesting pod/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 15:53:29.951710 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:29.951607    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 15:53:30.006642 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:30.006603    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:30.110177 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:30.110143    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:30.210341 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:30.210307    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:30.439615 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:30.439577    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:31.321732 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:31.321695    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:32.181562 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:32.181528    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:33.006403 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.006369    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:33.106309 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.106269    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:33.205631 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.205596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:33.557590 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.557556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:33.656597 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.656563    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:33.714351 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.714314    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"]
Sep 18 15:53:33.716910 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.716892    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"
Sep 18 15:53:33.720564 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.720545    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4e81aa-dockercfg-r69rc\""
Sep 18 15:53:33.720690 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.720586    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4e81aa\""
Sep 18 15:53:33.747041 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.747009    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"]
Sep 18 15:53:33.763940 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.763906    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"]
Sep 18 15:53:33.766765 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.766745    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"
Sep 18 15:53:33.791760 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.791726    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199bef39-11ee-4aca-bd23-778459f6b5e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m\" (UID: \"199bef39-11ee-4aca-bd23-778459f6b5e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"
Sep 18 15:53:33.791948 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.791920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvbrg\" (UniqueName: \"kubernetes.io/projected/199bef39-11ee-4aca-bd23-778459f6b5e0-kube-api-access-tvbrg\") pod \"payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m\" (UID: \"199bef39-11ee-4aca-bd23-778459f6b5e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"
Sep 18 15:53:33.799751 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.799719    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"]
Sep 18 15:53:33.892870 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.892782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbrg\" (UniqueName: \"kubernetes.io/projected/199bef39-11ee-4aca-bd23-778459f6b5e0-kube-api-access-tvbrg\") pod \"payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m\" (UID: \"199bef39-11ee-4aca-bd23-778459f6b5e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"
Sep 18 15:53:33.892870 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.892831    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qggz\" (UniqueName: \"kubernetes.io/projected/fca1c4b5-d927-4570-a16c-93de369c34f2-kube-api-access-4qggz\") pod \"payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx\" (UID: \"fca1c4b5-d927-4570-a16c-93de369c34f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"
Sep 18 15:53:33.893112 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.892938    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca1c4b5-d927-4570-a16c-93de369c34f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx\" (UID: \"fca1c4b5-d927-4570-a16c-93de369c34f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"
Sep 18 15:53:33.893112 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.892982    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199bef39-11ee-4aca-bd23-778459f6b5e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m\" (UID: \"199bef39-11ee-4aca-bd23-778459f6b5e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"
Sep 18 15:53:33.893657 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.893634    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199bef39-11ee-4aca-bd23-778459f6b5e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m\" (UID: \"199bef39-11ee-4aca-bd23-778459f6b5e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"
Sep 18 15:53:33.911179 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.911153    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvbrg\" (UniqueName: \"kubernetes.io/projected/199bef39-11ee-4aca-bd23-778459f6b5e0-kube-api-access-tvbrg\") pod \"payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m\" (UID: \"199bef39-11ee-4aca-bd23-778459f6b5e0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"
Sep 18 15:53:33.994187 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.994151    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qggz\" (UniqueName: \"kubernetes.io/projected/fca1c4b5-d927-4570-a16c-93de369c34f2-kube-api-access-4qggz\") pod \"payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx\" (UID: \"fca1c4b5-d927-4570-a16c-93de369c34f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"
Sep 18 15:53:33.994373 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.994220    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca1c4b5-d927-4570-a16c-93de369c34f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx\" (UID: \"fca1c4b5-d927-4570-a16c-93de369c34f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"
Sep 18 15:53:33.994928 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:33.994909    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca1c4b5-d927-4570-a16c-93de369c34f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx\" (UID: \"fca1c4b5-d927-4570-a16c-93de369c34f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"
Sep 18 15:53:34.003344 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.003312    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qggz\" (UniqueName: \"kubernetes.io/projected/fca1c4b5-d927-4570-a16c-93de369c34f2-kube-api-access-4qggz\") pod \"payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx\" (UID: \"fca1c4b5-d927-4570-a16c-93de369c34f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"
Sep 18 15:53:34.027882 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.027838    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"
Sep 18 15:53:34.077202 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.077115    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"
Sep 18 15:53:34.299903 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.299876    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"]
Sep 18 15:53:34.301037 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:53:34.301013    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199bef39_11ee_4aca_bd23_778459f6b5e0.slice/crio-3bb8a555bc47422a71592e33ca176345552ff63e78100c79e3d5f577426cbd6e WatchSource:0}: Error finding container 3bb8a555bc47422a71592e33ca176345552ff63e78100c79e3d5f577426cbd6e: Status 404 returned error can't find the container with id 3bb8a555bc47422a71592e33ca176345552ff63e78100c79e3d5f577426cbd6e
Sep 18 15:53:34.357798 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.357766    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"]
Sep 18 15:53:34.358750 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:53:34.358726    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca1c4b5_d927_4570_a16c_93de369c34f2.slice/crio-85fc318d36596df137368a466dd446945659f13833846c04339842a8fd9bbed4 WatchSource:0}: Error finding container 85fc318d36596df137368a466dd446945659f13833846c04339842a8fd9bbed4: Status 404 returned error can't find the container with id 85fc318d36596df137368a466dd446945659f13833846c04339842a8fd9bbed4
Sep 18 15:53:34.813429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.813402    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:53:34.902492 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.902411    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-token\") pod \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") "
Sep 18 15:53:34.902492 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.902446    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-credential-socket\") pod \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") "
Sep 18 15:53:34.902699 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.902516    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-workload-socket\") pod \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") "
Sep 18 15:53:34.902699 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.902541    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-podinfo\") pod \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") "
Sep 18 15:53:34.902699 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.902598    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-data\") pod \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") "
Sep 18 15:53:34.902699 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.902624    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-envoy\") pod \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") "
Sep 18 15:53:34.902699 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.902668    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istiod-ca-cert\") pod \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") "
Sep 18 15:53:34.902699 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.902697    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpzsj\" (UniqueName: \"kubernetes.io/projected/b1bee9c4-37ea-4909-8e44-af4bd98fb252-kube-api-access-kpzsj\") pod \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") "
Sep 18 15:53:34.903022 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.902747    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-workload-certs\") pod \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\" (UID: \"b1bee9c4-37ea-4909-8e44-af4bd98fb252\") "
Sep 18 15:53:34.903022 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.902806    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b1bee9c4-37ea-4909-8e44-af4bd98fb252" (UID: "b1bee9c4-37ea-4909-8e44-af4bd98fb252"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:53:34.903022 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.902814    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b1bee9c4-37ea-4909-8e44-af4bd98fb252" (UID: "b1bee9c4-37ea-4909-8e44-af4bd98fb252"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:53:34.903202 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.903071    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b1bee9c4-37ea-4909-8e44-af4bd98fb252" (UID: "b1bee9c4-37ea-4909-8e44-af4bd98fb252"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:53:34.903202 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.903109    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-credential-socket\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:34.903202 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.903125    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-workload-socket\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:34.903362 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.903226    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-data" (OuterVolumeSpecName: "istio-data") pod "b1bee9c4-37ea-4909-8e44-af4bd98fb252" (UID: "b1bee9c4-37ea-4909-8e44-af4bd98fb252"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:53:34.903362 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.903334    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b1bee9c4-37ea-4909-8e44-af4bd98fb252" (UID: "b1bee9c4-37ea-4909-8e44-af4bd98fb252"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:53:34.904776 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.904752    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-token" (OuterVolumeSpecName: "istio-token") pod "b1bee9c4-37ea-4909-8e44-af4bd98fb252" (UID: "b1bee9c4-37ea-4909-8e44-af4bd98fb252"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:53:34.904997 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.904971    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b1bee9c4-37ea-4909-8e44-af4bd98fb252" (UID: "b1bee9c4-37ea-4909-8e44-af4bd98fb252"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:53:34.905327 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.905307    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b1bee9c4-37ea-4909-8e44-af4bd98fb252" (UID: "b1bee9c4-37ea-4909-8e44-af4bd98fb252"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 15:53:34.905536 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.905518    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bee9c4-37ea-4909-8e44-af4bd98fb252-kube-api-access-kpzsj" (OuterVolumeSpecName: "kube-api-access-kpzsj") pod "b1bee9c4-37ea-4909-8e44-af4bd98fb252" (UID: "b1bee9c4-37ea-4909-8e44-af4bd98fb252"). InnerVolumeSpecName "kube-api-access-kpzsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:53:34.976248 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.976217    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m" event={"ID":"199bef39-11ee-4aca-bd23-778459f6b5e0","Type":"ContainerStarted","Data":"456a6f5ca48024743a2895eb9a5cb242590dec0d308dd365bdd3d0e1e24c229f"}
Sep 18 15:53:34.976424 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.976278    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m" event={"ID":"199bef39-11ee-4aca-bd23-778459f6b5e0","Type":"ContainerStarted","Data":"3bb8a555bc47422a71592e33ca176345552ff63e78100c79e3d5f577426cbd6e"}
Sep 18 15:53:34.976424 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.976307    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"
Sep 18 15:53:34.977919 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.977896    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx" event={"ID":"fca1c4b5-d927-4570-a16c-93de369c34f2","Type":"ContainerStarted","Data":"b996dbc34f622d70f852e6a225387eeab0175407b966e3f99d6c0c87ee074656"}
Sep 18 15:53:34.978063 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.977944    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx" event={"ID":"fca1c4b5-d927-4570-a16c-93de369c34f2","Type":"ContainerStarted","Data":"85fc318d36596df137368a466dd446945659f13833846c04339842a8fd9bbed4"}
Sep 18 15:53:34.978063 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.977973    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"
Sep 18 15:53:34.983061 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.983016    2559 generic.go:356] "Generic (PLEG): container finished" podID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerID="dba98aad75549ccf74282e42d9c3182a449e0c37aa7f8fce7d355447ac60199f" exitCode=0
Sep 18 15:53:34.983201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.983063    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"
Sep 18 15:53:34.983201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.983135    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" event={"ID":"b1bee9c4-37ea-4909-8e44-af4bd98fb252","Type":"ContainerDied","Data":"dba98aad75549ccf74282e42d9c3182a449e0c37aa7f8fce7d355447ac60199f"}
Sep 18 15:53:34.983201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.983177    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq" event={"ID":"b1bee9c4-37ea-4909-8e44-af4bd98fb252","Type":"ContainerDied","Data":"09942d22ae0a47b0c798b222a0b6e4298884b59e228b2c54cf1cd78dab561742"}
Sep 18 15:53:34.983366 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.983206    2559 scope.go:117] "RemoveContainer" containerID="dba98aad75549ccf74282e42d9c3182a449e0c37aa7f8fce7d355447ac60199f"
Sep 18 15:53:34.993418 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.993401    2559 scope.go:117] "RemoveContainer" containerID="f2966897a5b54e63f77b765fb9152e2013f75e7c4e307d8335578004740dd816"
Sep 18 15:53:34.997725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:34.997678    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m" podStartSLOduration=1.997660014 podStartE2EDuration="1.997660014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:53:34.996533954 +0000 UTC m=+1137.557766096" watchObservedRunningTime="2026-09-18 15:53:34.997660014 +0000 UTC m=+1137.558892151"
Sep 18 15:53:35.002354 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.002333    2559 scope.go:117] "RemoveContainer" containerID="dba98aad75549ccf74282e42d9c3182a449e0c37aa7f8fce7d355447ac60199f"
Sep 18 15:53:35.002662 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:53:35.002642    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba98aad75549ccf74282e42d9c3182a449e0c37aa7f8fce7d355447ac60199f\": container with ID starting with dba98aad75549ccf74282e42d9c3182a449e0c37aa7f8fce7d355447ac60199f not found: ID does not exist" containerID="dba98aad75549ccf74282e42d9c3182a449e0c37aa7f8fce7d355447ac60199f"
Sep 18 15:53:35.002749 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.002674    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba98aad75549ccf74282e42d9c3182a449e0c37aa7f8fce7d355447ac60199f"} err="failed to get container status \"dba98aad75549ccf74282e42d9c3182a449e0c37aa7f8fce7d355447ac60199f\": rpc error: code = NotFound desc = could not find container \"dba98aad75549ccf74282e42d9c3182a449e0c37aa7f8fce7d355447ac60199f\": container with ID starting with dba98aad75549ccf74282e42d9c3182a449e0c37aa7f8fce7d355447ac60199f not found: ID does not exist"
Sep 18 15:53:35.002749 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.002697    2559 scope.go:117] "RemoveContainer" containerID="f2966897a5b54e63f77b765fb9152e2013f75e7c4e307d8335578004740dd816"
Sep 18 15:53:35.002988 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:53:35.002969    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2966897a5b54e63f77b765fb9152e2013f75e7c4e307d8335578004740dd816\": container with ID starting with f2966897a5b54e63f77b765fb9152e2013f75e7c4e307d8335578004740dd816 not found: ID does not exist" containerID="f2966897a5b54e63f77b765fb9152e2013f75e7c4e307d8335578004740dd816"
Sep 18 15:53:35.003054 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.002995    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2966897a5b54e63f77b765fb9152e2013f75e7c4e307d8335578004740dd816"} err="failed to get container status \"f2966897a5b54e63f77b765fb9152e2013f75e7c4e307d8335578004740dd816\": rpc error: code = NotFound desc = could not find container \"f2966897a5b54e63f77b765fb9152e2013f75e7c4e307d8335578004740dd816\": container with ID starting with f2966897a5b54e63f77b765fb9152e2013f75e7c4e307d8335578004740dd816 not found: ID does not exist"
Sep 18 15:53:35.004490 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.004471    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-workload-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:35.004540 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.004496    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-token\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:35.004540 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.004509    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-podinfo\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:35.004540 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.004521    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-data\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:35.004540 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.004533    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istio-envoy\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:35.004737 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.004545    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1bee9c4-37ea-4909-8e44-af4bd98fb252-istiod-ca-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:35.004737 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.004558    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpzsj\" (UniqueName: \"kubernetes.io/projected/b1bee9c4-37ea-4909-8e44-af4bd98fb252-kube-api-access-kpzsj\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:53:35.012601 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.012579    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"]
Sep 18 15:53:35.015670 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.015650    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f84020ef-openshift-default-569b5b6c96-4vzlq"]
Sep 18 15:53:35.032246 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.032208    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx" podStartSLOduration=2.032195187 podStartE2EDuration="2.032195187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:53:35.030524253 +0000 UTC m=+1137.591756393" watchObservedRunningTime="2026-09-18 15:53:35.032195187 +0000 UTC m=+1137.593427327"
Sep 18 15:53:35.745028 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:35.744995    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" path="/var/lib/kubelet/pods/b1bee9c4-37ea-4909-8e44-af4bd98fb252/volumes"
Sep 18 15:53:36.190703 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:36.190678    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:53:36.395558 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:36.395522    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:36.506526 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:36.506434    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:36.637939 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:36.637902    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:36.732822 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:36.732789    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:38.237477 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:38.237445    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:41.592897 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:41.592842    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:42.550138 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:42.550105    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:42.659672 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:42.659641    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:43.496565 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:43.496531    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:43.798439 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:43.798347    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:44.372610 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:44.372575    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:45.909068 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:45.909032    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:45.992662 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:45.992637    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"
Sep 18 15:53:45.992868 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:45.992702    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"
Sep 18 15:53:46.678729 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:46.678690    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:46.784893 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:46.784831    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:47.757018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:47.756980    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:48.735323 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:48.735286    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:49.764572 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:49.764542    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:49.869728 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:49.869693    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:50.688467 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:50.688429    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:50.759098 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:50.759060    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:51.033507 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.033414    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:51.282355 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.282276    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:51.379414 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.379325    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:51.385170 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.385141    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"]
Sep 18 15:53:51.385572 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.385561    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy"
Sep 18 15:53:51.385616 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.385575    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy"
Sep 18 15:53:51.385653 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.385643    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy"
Sep 18 15:53:51.385687 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.385654    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy"
Sep 18 15:53:51.388954 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.388931    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"
Sep 18 15:53:51.393172 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.393155    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-195881\""
Sep 18 15:53:51.393324 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.393154    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-195881-dockercfg-4nmnj\""
Sep 18 15:53:51.399786 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.399761    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"]
Sep 18 15:53:51.414907 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.414885    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"]
Sep 18 15:53:51.415282 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.415271    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy"
Sep 18 15:53:51.415325 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.415283    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bee9c4-37ea-4909-8e44-af4bd98fb252" containerName="istio-proxy"
Sep 18 15:53:51.417422 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.417406    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"
Sep 18 15:53:51.429236 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.429209    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"]
Sep 18 15:53:51.455504 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.455479    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7g4q\" (UniqueName: \"kubernetes.io/projected/732529c0-3590-4c3f-b1ae-13a1cdf477b4-kube-api-access-k7g4q\") pod \"payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226\" (UID: \"732529c0-3590-4c3f-b1ae-13a1cdf477b4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"
Sep 18 15:53:51.455627 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.455525    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e9df99-8c7d-4055-a1de-59a80d25ea41-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9\" (UID: \"58e9df99-8c7d-4055-a1de-59a80d25ea41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"
Sep 18 15:53:51.455678 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.455632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732529c0-3590-4c3f-b1ae-13a1cdf477b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226\" (UID: \"732529c0-3590-4c3f-b1ae-13a1cdf477b4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"
Sep 18 15:53:51.455678 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.455666    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-925k5\" (UniqueName: \"kubernetes.io/projected/58e9df99-8c7d-4055-a1de-59a80d25ea41-kube-api-access-925k5\") pod \"payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9\" (UID: \"58e9df99-8c7d-4055-a1de-59a80d25ea41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"
Sep 18 15:53:51.556106 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.556073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e9df99-8c7d-4055-a1de-59a80d25ea41-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9\" (UID: \"58e9df99-8c7d-4055-a1de-59a80d25ea41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"
Sep 18 15:53:51.556281 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.556126    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732529c0-3590-4c3f-b1ae-13a1cdf477b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226\" (UID: \"732529c0-3590-4c3f-b1ae-13a1cdf477b4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"
Sep 18 15:53:51.556332 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.556288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-925k5\" (UniqueName: \"kubernetes.io/projected/58e9df99-8c7d-4055-a1de-59a80d25ea41-kube-api-access-925k5\") pod \"payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9\" (UID: \"58e9df99-8c7d-4055-a1de-59a80d25ea41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"
Sep 18 15:53:51.556389 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.556377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7g4q\" (UniqueName: \"kubernetes.io/projected/732529c0-3590-4c3f-b1ae-13a1cdf477b4-kube-api-access-k7g4q\") pod \"payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226\" (UID: \"732529c0-3590-4c3f-b1ae-13a1cdf477b4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"
Sep 18 15:53:51.556799 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.556779    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e9df99-8c7d-4055-a1de-59a80d25ea41-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9\" (UID: \"58e9df99-8c7d-4055-a1de-59a80d25ea41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"
Sep 18 15:53:51.556976 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.556957    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732529c0-3590-4c3f-b1ae-13a1cdf477b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226\" (UID: \"732529c0-3590-4c3f-b1ae-13a1cdf477b4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"
Sep 18 15:53:51.564614 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.564583    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-925k5\" (UniqueName: \"kubernetes.io/projected/58e9df99-8c7d-4055-a1de-59a80d25ea41-kube-api-access-925k5\") pod \"payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9\" (UID: \"58e9df99-8c7d-4055-a1de-59a80d25ea41\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"
Sep 18 15:53:51.564809 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.564788    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7g4q\" (UniqueName: \"kubernetes.io/projected/732529c0-3590-4c3f-b1ae-13a1cdf477b4-kube-api-access-k7g4q\") pod \"payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226\" (UID: \"732529c0-3590-4c3f-b1ae-13a1cdf477b4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"
Sep 18 15:53:51.677010 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.676921    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:51.700793 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.700766    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"
Sep 18 15:53:51.726886 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.726850    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"
Sep 18 15:53:51.979561 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:51.979535    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"]
Sep 18 15:53:51.981606 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:53:51.981582    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e9df99_8c7d_4055_a1de_59a80d25ea41.slice/crio-2b85d7088d213bad167e8dbf89fd8ff53a3d0b0e235dbcec39d1c3d5ce6046bc WatchSource:0}: Error finding container 2b85d7088d213bad167e8dbf89fd8ff53a3d0b0e235dbcec39d1c3d5ce6046bc: Status 404 returned error can't find the container with id 2b85d7088d213bad167e8dbf89fd8ff53a3d0b0e235dbcec39d1c3d5ce6046bc
Sep 18 15:53:52.015615 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:52.015590    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"]
Sep 18 15:53:52.019927 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:53:52.019904    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732529c0_3590_4c3f_b1ae_13a1cdf477b4.slice/crio-7d914971de09bbfc8ce8337bee564ccb6ed1b98fe45755022819aa4d596851de WatchSource:0}: Error finding container 7d914971de09bbfc8ce8337bee564ccb6ed1b98fe45755022819aa4d596851de: Status 404 returned error can't find the container with id 7d914971de09bbfc8ce8337bee564ccb6ed1b98fe45755022819aa4d596851de
Sep 18 15:53:52.078634 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:52.078599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9" event={"ID":"58e9df99-8c7d-4055-a1de-59a80d25ea41","Type":"ContainerStarted","Data":"6772eb3c1cfcefd7f5bc334a65f7f9a5ef40ad3282dcc64ed02712dbe85159ab"}
Sep 18 15:53:52.079094 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:52.078665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9" event={"ID":"58e9df99-8c7d-4055-a1de-59a80d25ea41","Type":"ContainerStarted","Data":"2b85d7088d213bad167e8dbf89fd8ff53a3d0b0e235dbcec39d1c3d5ce6046bc"}
Sep 18 15:53:52.079094 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:52.078792    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"
Sep 18 15:53:52.080494 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:52.080469    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226" event={"ID":"732529c0-3590-4c3f-b1ae-13a1cdf477b4","Type":"ContainerStarted","Data":"c4cc69f3efefc63516575103f254880454a030357a5a7dc48726f6bc9746c452"}
Sep 18 15:53:52.080618 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:52.080516    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226" event={"ID":"732529c0-3590-4c3f-b1ae-13a1cdf477b4","Type":"ContainerStarted","Data":"7d914971de09bbfc8ce8337bee564ccb6ed1b98fe45755022819aa4d596851de"}
Sep 18 15:53:52.080618 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:52.080570    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"
Sep 18 15:53:52.096391 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:52.096336    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9" podStartSLOduration=1.096322477 podStartE2EDuration="1.096322477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:53:52.093800215 +0000 UTC m=+1154.655032406" watchObservedRunningTime="2026-09-18 15:53:52.096322477 +0000 UTC m=+1154.657554616"
Sep 18 15:53:52.115085 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:52.115027    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226" podStartSLOduration=1.115007407 podStartE2EDuration="1.115007407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:53:52.113031506 +0000 UTC m=+1154.674263646" watchObservedRunningTime="2026-09-18 15:53:52.115007407 +0000 UTC m=+1154.676239548"
Sep 18 15:53:52.412533 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:52.412434    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:53.219559 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:53.219516    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:54.176551 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:54.176515    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:54.907381 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:54.907342    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:53:55.008663 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:53:55.008624    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:03.087629 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:03.087602    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"
Sep 18 15:54:03.093905 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:03.087687    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"
Sep 18 15:54:04.145765 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:04.145732    2559 generic.go:356] "Generic (PLEG): container finished" podID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerID="66398535e44629b3fb3c976c20ade87e2d358d896e67d1587b556c3cd4d51053" exitCode=0
Sep 18 15:54:04.145765 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:04.145775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" event={"ID":"af9fcf83-d452-4650-bbcd-a5d13a7e8884","Type":"ContainerDied","Data":"66398535e44629b3fb3c976c20ade87e2d358d896e67d1587b556c3cd4d51053"}
Sep 18 15:54:04.146350 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:04.146263    2559 scope.go:117] "RemoveContainer" containerID="66398535e44629b3fb3c976c20ade87e2d358d896e67d1587b556c3cd4d51053"
Sep 18 15:54:04.147431 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:04.147400    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:54:04.147487 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:04.147475    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:54:04.147522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:04.147507    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 15:54:05.153380 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:05.153350    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" event={"ID":"af9fcf83-d452-4650-bbcd-a5d13a7e8884","Type":"ContainerStarted","Data":"b792b8d3e9bf44350a7c12f5f51df2affba39cb7dcd628d2def381831918a4db"}
Sep 18 15:54:06.065398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:06.065361    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:54:06.065814 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:06.065783    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 18 15:54:06.066110 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:06.066057    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 18 15:54:07.065462 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:07.065425    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 18 15:54:07.066066 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:07.066021    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 18 15:54:08.065027 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:08.064978    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 18 15:54:08.065258 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:08.065052    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 18 15:54:08.283993 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:08.283951    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:54:09.065536 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:09.065103    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 18 15:54:09.065536 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:09.065166    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 18 15:54:10.064742 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:10.064703    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 18 15:54:10.065168 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:10.064761    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 18 15:54:11.069096 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:11.069056    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:54:11.069565 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:11.069476    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:54:11.070283 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:11.070264    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 15:54:11.783720 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:11.783687    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:13.195539 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:13.195508    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:13.930887 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:13.930833    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:14.791463 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:14.791418    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:15.401588 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:15.401553    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:16.113423 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:16.113381    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:16.613511 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:16.613472    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:16.918186 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:16.918107    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:17.053633 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:17.053599    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:17.313616 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:17.313580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:17.639690 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:17.639600    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:19.598468 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:19.598432    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:20.317037 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:20.316995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:20.437651 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:20.437609    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:20.558192 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:20.558157    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:21.224046 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:21.224013    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:21.434175 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:21.434137    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:22.370099 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:22.370044    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:23.385695 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:23.385656    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:23.965885 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:23.965837    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:24.806050 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:24.806015    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:24.913732 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:24.913693    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:25.010646 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.010612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:25.422237 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.422203    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:25.519725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.519686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:25.530677 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.530647    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"]
Sep 18 15:54:25.533959 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.533941    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"
Sep 18 15:54:25.536771 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.536750    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-655d9c\""
Sep 18 15:54:25.547075 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.547054    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"]
Sep 18 15:54:25.584914 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.584888    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfab3c15-ce3e-40b1-a9af-ca81282a5a2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl\" (UID: \"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"
Sep 18 15:54:25.585050 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.584936    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86v56\" (UniqueName: \"kubernetes.io/projected/dfab3c15-ce3e-40b1-a9af-ca81282a5a2e-kube-api-access-86v56\") pod \"payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl\" (UID: \"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"
Sep 18 15:54:25.669518 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.669483    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"]
Sep 18 15:54:25.672583 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.672536    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"
Sep 18 15:54:25.676110 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.676086    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-655d9c-dockercfg-5p8tn\""
Sep 18 15:54:25.685690 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.685664    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4fpq\" (UniqueName: \"kubernetes.io/projected/3e51525f-2ffb-46ae-90fa-ba7bee1a7832-kube-api-access-p4fpq\") pod \"payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb\" (UID: \"3e51525f-2ffb-46ae-90fa-ba7bee1a7832\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"
Sep 18 15:54:25.685828 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.685732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfab3c15-ce3e-40b1-a9af-ca81282a5a2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl\" (UID: \"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"
Sep 18 15:54:25.685828 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.685795    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86v56\" (UniqueName: \"kubernetes.io/projected/dfab3c15-ce3e-40b1-a9af-ca81282a5a2e-kube-api-access-86v56\") pod \"payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl\" (UID: \"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"
Sep 18 15:54:25.685986 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.685971    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e51525f-2ffb-46ae-90fa-ba7bee1a7832-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb\" (UID: \"3e51525f-2ffb-46ae-90fa-ba7bee1a7832\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"
Sep 18 15:54:25.686573 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.686550    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfab3c15-ce3e-40b1-a9af-ca81282a5a2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl\" (UID: \"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"
Sep 18 15:54:25.689594 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.689568    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"]
Sep 18 15:54:25.706910 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.706846    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86v56\" (UniqueName: \"kubernetes.io/projected/dfab3c15-ce3e-40b1-a9af-ca81282a5a2e-kube-api-access-86v56\") pod \"payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl\" (UID: \"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"
Sep 18 15:54:25.786847 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.786821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e51525f-2ffb-46ae-90fa-ba7bee1a7832-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb\" (UID: \"3e51525f-2ffb-46ae-90fa-ba7bee1a7832\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"
Sep 18 15:54:25.787070 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.787043    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4fpq\" (UniqueName: \"kubernetes.io/projected/3e51525f-2ffb-46ae-90fa-ba7bee1a7832-kube-api-access-p4fpq\") pod \"payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb\" (UID: \"3e51525f-2ffb-46ae-90fa-ba7bee1a7832\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"
Sep 18 15:54:25.787579 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.787559    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e51525f-2ffb-46ae-90fa-ba7bee1a7832-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb\" (UID: \"3e51525f-2ffb-46ae-90fa-ba7bee1a7832\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"
Sep 18 15:54:25.796976 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.796956    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4fpq\" (UniqueName: \"kubernetes.io/projected/3e51525f-2ffb-46ae-90fa-ba7bee1a7832-kube-api-access-p4fpq\") pod \"payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb\" (UID: \"3e51525f-2ffb-46ae-90fa-ba7bee1a7832\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"
Sep 18 15:54:25.844804 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.844782    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"
Sep 18 15:54:25.985386 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:25.985357    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"
Sep 18 15:54:26.113957 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:26.113931    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"]
Sep 18 15:54:26.115413 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:54:26.115385    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfab3c15_ce3e_40b1_a9af_ca81282a5a2e.slice/crio-75abf22f78750c8eed8a7d3be3cfd5f2976c1acfb5fa895fd87923ef37ac40af WatchSource:0}: Error finding container 75abf22f78750c8eed8a7d3be3cfd5f2976c1acfb5fa895fd87923ef37ac40af: Status 404 returned error can't find the container with id 75abf22f78750c8eed8a7d3be3cfd5f2976c1acfb5fa895fd87923ef37ac40af
Sep 18 15:54:26.259253 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:26.259221    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"]
Sep 18 15:54:26.261423 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:54:26.261390    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e51525f_2ffb_46ae_90fa_ba7bee1a7832.slice/crio-f833388c357189fa36c4cc0cf9bd81e6939f8f2ed76e8c4d5cd553c8e7607cf0 WatchSource:0}: Error finding container f833388c357189fa36c4cc0cf9bd81e6939f8f2ed76e8c4d5cd553c8e7607cf0: Status 404 returned error can't find the container with id f833388c357189fa36c4cc0cf9bd81e6939f8f2ed76e8c4d5cd553c8e7607cf0
Sep 18 15:54:26.278300 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:26.278276    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb" event={"ID":"3e51525f-2ffb-46ae-90fa-ba7bee1a7832","Type":"ContainerStarted","Data":"f833388c357189fa36c4cc0cf9bd81e6939f8f2ed76e8c4d5cd553c8e7607cf0"}
Sep 18 15:54:26.280202 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:26.280175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl" event={"ID":"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e","Type":"ContainerStarted","Data":"914e0529d5b145f2336ba4a109617ed72c90c3de99e76903324467792f4f8208"}
Sep 18 15:54:26.280311 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:26.280228    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl" event={"ID":"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e","Type":"ContainerStarted","Data":"75abf22f78750c8eed8a7d3be3cfd5f2976c1acfb5fa895fd87923ef37ac40af"}
Sep 18 15:54:26.280311 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:26.280272    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"
Sep 18 15:54:26.297476 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:26.297434    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl" podStartSLOduration=1.297421912 podStartE2EDuration="1.297421912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:54:26.296082437 +0000 UTC m=+1188.857314576" watchObservedRunningTime="2026-09-18 15:54:26.297421912 +0000 UTC m=+1188.858654046"
Sep 18 15:54:27.288553 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:27.288512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb" event={"ID":"3e51525f-2ffb-46ae-90fa-ba7bee1a7832","Type":"ContainerStarted","Data":"b11b284e9525d4c4f683320feddae6cd98a22a5aad2953c3ff981161f8476cd3"}
Sep 18 15:54:27.307257 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:27.307202    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb" podStartSLOduration=2.307181954 podStartE2EDuration="2.307181954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:54:27.306458635 +0000 UTC m=+1189.867690776" watchObservedRunningTime="2026-09-18 15:54:27.307181954 +0000 UTC m=+1189.868414098"
Sep 18 15:54:28.294771 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:28.294736    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"
Sep 18 15:54:37.290841 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:37.290806    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"
Sep 18 15:54:39.302133 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:39.302105    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"
Sep 18 15:54:40.515622 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:40.515586    2559 scope.go:117] "RemoveContainer" containerID="1725f73fc1822296a7e40c3767cab0e70920cc513421ece7180f881f521b1ad9"
Sep 18 15:54:47.173252 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:47.173214    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:48.006722 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:48.006682    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:48.782012 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:48.781974    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:54:48.902647 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:48.902595    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:49.652980 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:49.652940    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:50.294877 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:50.294814    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:50.474994 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:50.474941    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:51.352733 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:51.352696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:52.210532 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:52.210495    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:55.477816 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:55.477777    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:55.595448 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:55.595418    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj"]
Sep 18 15:54:55.602893 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:55.602467    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj"
Sep 18 15:54:55.605791 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:55.605767    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 15:54:55.605954 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:55.605845    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 15:54:55.607144 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:55.607100    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj"]
Sep 18 15:54:55.777428 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:55.777349    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b701640-bd1c-4fba-98b7-50be967fc959-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj\" (UID: \"5b701640-bd1c-4fba-98b7-50be967fc959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj"
Sep 18 15:54:55.878216 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:55.878181    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b701640-bd1c-4fba-98b7-50be967fc959-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj\" (UID: \"5b701640-bd1c-4fba-98b7-50be967fc959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj"
Sep 18 15:54:55.878809 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:55.878787    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b701640-bd1c-4fba-98b7-50be967fc959-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj\" (UID: \"5b701640-bd1c-4fba-98b7-50be967fc959\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj"
Sep 18 15:54:55.914493 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:55.914465    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj"
Sep 18 15:54:56.079520 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:56.079430    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:56.185725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:56.185696    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj"]
Sep 18 15:54:56.187294 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:54:56.187269    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b701640_bd1c_4fba_98b7_50be967fc959.slice/crio-27df2b81f7222db5b46401c7ad12d765f9b0004ac2020f02caf4705503e6750b WatchSource:0}: Error finding container 27df2b81f7222db5b46401c7ad12d765f9b0004ac2020f02caf4705503e6750b: Status 404 returned error can't find the container with id 27df2b81f7222db5b46401c7ad12d765f9b0004ac2020f02caf4705503e6750b
Sep 18 15:54:56.453839 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:56.453798    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj" event={"ID":"5b701640-bd1c-4fba-98b7-50be967fc959","Type":"ContainerStarted","Data":"27df2b81f7222db5b46401c7ad12d765f9b0004ac2020f02caf4705503e6750b"}
Sep 18 15:54:56.854947 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:56.854886    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:57.460998 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:57.460968    2559 generic.go:356] "Generic (PLEG): container finished" podID="5b701640-bd1c-4fba-98b7-50be967fc959" containerID="bf9d170c43d2fa31f0de8d2ffa95a26b3e5c802cf2ea4fae9f4e37697b47b502" exitCode=0
Sep 18 15:54:57.461177 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:57.461068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj" event={"ID":"5b701640-bd1c-4fba-98b7-50be967fc959","Type":"ContainerDied","Data":"bf9d170c43d2fa31f0de8d2ffa95a26b3e5c802cf2ea4fae9f4e37697b47b502"}
Sep 18 15:54:57.639832 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:57.639790    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:58.359772 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:58.359713    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:58.610233 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:58.610177    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj"
Sep 18 15:54:58.806620 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:58.806591    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b701640-bd1c-4fba-98b7-50be967fc959-maas-api-service-ca\") pod \"5b701640-bd1c-4fba-98b7-50be967fc959\" (UID: \"5b701640-bd1c-4fba-98b7-50be967fc959\") "
Sep 18 15:54:58.807032 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:58.807007    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b701640-bd1c-4fba-98b7-50be967fc959-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b701640-bd1c-4fba-98b7-50be967fc959" (UID: "5b701640-bd1c-4fba-98b7-50be967fc959"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:54:58.908190 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:58.908086    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b701640-bd1c-4fba-98b7-50be967fc959-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:54:59.188158 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:59.188117    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:59.473893 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:59.473787    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj"
Sep 18 15:54:59.473893 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:59.473784    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj" event={"ID":"5b701640-bd1c-4fba-98b7-50be967fc959","Type":"ContainerDied","Data":"27df2b81f7222db5b46401c7ad12d765f9b0004ac2020f02caf4705503e6750b"}
Sep 18 15:54:59.474348 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:59.473934    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27df2b81f7222db5b46401c7ad12d765f9b0004ac2020f02caf4705503e6750b"
Sep 18 15:54:59.774349 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:59.774275    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:54:59.917911 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:54:59.917881    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:00.072493 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:00.072399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:00.261253 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:00.261221    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:00.460586 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:00.460555    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"]
Sep 18 15:55:00.489526 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:00.489497    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"]
Sep 18 15:55:00.577584 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:00.577548    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:01.190405 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:01.190367    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:02.013040 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:02.012999    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:03.123171 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:03.123136    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:03.240582 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:03.240547    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:04.544357 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:04.544316    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:05.794687 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:05.794651    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:06.810323 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:06.810278    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:07.580557 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:07.580519    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:08.255465 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:08.255424    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:09.295509 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:09.295471    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:09.404283 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:09.404248    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:09.591884 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:09.591780    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:09.898657 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:09.898543    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:09.998502 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:09.998464    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:10.014937 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.014905    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"]
Sep 18 15:55:10.015335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.015323    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b701640-bd1c-4fba-98b7-50be967fc959" containerName="revoke-keys"
Sep 18 15:55:10.015381 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.015338    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b701640-bd1c-4fba-98b7-50be967fc959" containerName="revoke-keys"
Sep 18 15:55:10.015424 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.015416    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b701640-bd1c-4fba-98b7-50be967fc959" containerName="revoke-keys"
Sep 18 15:55:10.018751 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.018729    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"
Sep 18 15:55:10.021567 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.021547    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c56bfef6\""
Sep 18 15:55:10.021693 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.021669    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c56bfef6-dockercfg-9mq59\""
Sep 18 15:55:10.031091 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.031067    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"]
Sep 18 15:55:10.058136 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.055103    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"]
Sep 18 15:55:10.060555 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.060531    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"
Sep 18 15:55:10.064498 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.064468    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"]
Sep 18 15:55:10.099560 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.099531    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxdd\" (UniqueName: \"kubernetes.io/projected/1ca50fd5-9d55-48d6-aee1-05cd69881bc7-kube-api-access-tpxdd\") pod \"payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z\" (UID: \"1ca50fd5-9d55-48d6-aee1-05cd69881bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"
Sep 18 15:55:10.099713 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.099572    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca50fd5-9d55-48d6-aee1-05cd69881bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z\" (UID: \"1ca50fd5-9d55-48d6-aee1-05cd69881bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"
Sep 18 15:55:10.099713 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.099613    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25faaa2-65e2-4255-8966-843f9482c1c6-plugins-config-volume\") pod \"payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj\" (UID: \"a25faaa2-65e2-4255-8966-843f9482c1c6\") " pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"
Sep 18 15:55:10.099713 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.099681    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxmjh\" (UniqueName: \"kubernetes.io/projected/a25faaa2-65e2-4255-8966-843f9482c1c6-kube-api-access-vxmjh\") pod \"payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj\" (UID: \"a25faaa2-65e2-4255-8966-843f9482c1c6\") " pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"
Sep 18 15:55:10.200964 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.200924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxdd\" (UniqueName: \"kubernetes.io/projected/1ca50fd5-9d55-48d6-aee1-05cd69881bc7-kube-api-access-tpxdd\") pod \"payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z\" (UID: \"1ca50fd5-9d55-48d6-aee1-05cd69881bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"
Sep 18 15:55:10.201141 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.200970    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca50fd5-9d55-48d6-aee1-05cd69881bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z\" (UID: \"1ca50fd5-9d55-48d6-aee1-05cd69881bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"
Sep 18 15:55:10.201141 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.201013    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25faaa2-65e2-4255-8966-843f9482c1c6-plugins-config-volume\") pod \"payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj\" (UID: \"a25faaa2-65e2-4255-8966-843f9482c1c6\") " pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"
Sep 18 15:55:10.201141 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.201092    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxmjh\" (UniqueName: \"kubernetes.io/projected/a25faaa2-65e2-4255-8966-843f9482c1c6-kube-api-access-vxmjh\") pod \"payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj\" (UID: \"a25faaa2-65e2-4255-8966-843f9482c1c6\") " pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"
Sep 18 15:55:10.201841 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.201818    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca50fd5-9d55-48d6-aee1-05cd69881bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z\" (UID: \"1ca50fd5-9d55-48d6-aee1-05cd69881bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"
Sep 18 15:55:10.202011 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.201818    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25faaa2-65e2-4255-8966-843f9482c1c6-plugins-config-volume\") pod \"payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj\" (UID: \"a25faaa2-65e2-4255-8966-843f9482c1c6\") " pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"
Sep 18 15:55:10.209367 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.209342    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxdd\" (UniqueName: \"kubernetes.io/projected/1ca50fd5-9d55-48d6-aee1-05cd69881bc7-kube-api-access-tpxdd\") pod \"payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z\" (UID: \"1ca50fd5-9d55-48d6-aee1-05cd69881bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"
Sep 18 15:55:10.209471 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.209349    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxmjh\" (UniqueName: \"kubernetes.io/projected/a25faaa2-65e2-4255-8966-843f9482c1c6-kube-api-access-vxmjh\") pod \"payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj\" (UID: \"a25faaa2-65e2-4255-8966-843f9482c1c6\") " pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"
Sep 18 15:55:10.330185 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.330147    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"
Sep 18 15:55:10.372932 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.372903    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"
Sep 18 15:55:10.612559 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.612530    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"]
Sep 18 15:55:10.613659 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:55:10.613634    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca50fd5_9d55_48d6_aee1_05cd69881bc7.slice/crio-d006577799e67f5eab01f26d4920ece9507ea079287c898e622ddbbf89b5099e WatchSource:0}: Error finding container d006577799e67f5eab01f26d4920ece9507ea079287c898e622ddbbf89b5099e: Status 404 returned error can't find the container with id d006577799e67f5eab01f26d4920ece9507ea079287c898e622ddbbf89b5099e
Sep 18 15:55:10.662547 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:10.662521    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"]
Sep 18 15:55:10.668034 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:55:10.668007    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25faaa2_65e2_4255_8966_843f9482c1c6.slice/crio-b8aa33af76ea3146e053dbee6a0dc26730b3bf404efbc7e54d23ff11d0ec5e56 WatchSource:0}: Error finding container b8aa33af76ea3146e053dbee6a0dc26730b3bf404efbc7e54d23ff11d0ec5e56: Status 404 returned error can't find the container with id b8aa33af76ea3146e053dbee6a0dc26730b3bf404efbc7e54d23ff11d0ec5e56
Sep 18 15:55:11.551572 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:11.551534    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj" event={"ID":"a25faaa2-65e2-4255-8966-843f9482c1c6","Type":"ContainerStarted","Data":"3f66f698277e8ed98cf981d8235871d71c1d55a7ef4488925a2412dde4a79385"}
Sep 18 15:55:11.552050 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:11.551625    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj" event={"ID":"a25faaa2-65e2-4255-8966-843f9482c1c6","Type":"ContainerStarted","Data":"b8aa33af76ea3146e053dbee6a0dc26730b3bf404efbc7e54d23ff11d0ec5e56"}
Sep 18 15:55:11.552050 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:11.551661    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"
Sep 18 15:55:11.553267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:11.553233    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z" event={"ID":"1ca50fd5-9d55-48d6-aee1-05cd69881bc7","Type":"ContainerStarted","Data":"3f9b5352016b1db1ee4653e7a416ce73f7d28763e0773437cf9e4eeba9045861"}
Sep 18 15:55:11.553398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:11.553282    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z" event={"ID":"1ca50fd5-9d55-48d6-aee1-05cd69881bc7","Type":"ContainerStarted","Data":"d006577799e67f5eab01f26d4920ece9507ea079287c898e622ddbbf89b5099e"}
Sep 18 15:55:11.553398 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:11.553303    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"
Sep 18 15:55:11.569654 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:11.569606    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj" podStartSLOduration=1.569592683 podStartE2EDuration="1.569592683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:55:11.568315121 +0000 UTC m=+1234.129547286" watchObservedRunningTime="2026-09-18 15:55:11.569592683 +0000 UTC m=+1234.130824823"
Sep 18 15:55:11.585689 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:11.585629    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z" podStartSLOduration=2.585616977 podStartE2EDuration="2.585616977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:55:11.584090669 +0000 UTC m=+1234.145322811" watchObservedRunningTime="2026-09-18 15:55:11.585616977 +0000 UTC m=+1234.146849114"
Sep 18 15:55:14.153417 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:14.153370    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:14.256427 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:14.256386    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:14.355738 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:14.355702    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:14.407110 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:14.407020    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj"]
Sep 18 15:55:14.414912 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:14.414877    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9792ff-5tbpj"]
Sep 18 15:55:14.453894 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:14.453827    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:14.745868 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:14.745823    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:14.903565 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:14.903524    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:15.481734 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.481683    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h" podUID="a59feb5c-d29c-4281-a2ba-afd314a237d0" containerName="main" containerID="cri-o://b6288d13088e3a9f6295bbf201636c31625206765305256826ead8b6647908f0" gracePeriod=45
Sep 18 15:55:15.508677 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.508639    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw" podUID="5c8e57a6-40c3-4530-83b9-8b4c502347af" containerName="main" containerID="cri-o://f36d8dbf7c9131657d9923ba66c1cf5e3799bb1b0301be0761e228767c6e1d4a" gracePeriod=45
Sep 18 15:55:15.582333 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.582291    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:15.756351 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.756268    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b701640-bd1c-4fba-98b7-50be967fc959" path="/var/lib/kubelet/pods/5b701640-bd1c-4fba-98b7-50be967fc959/volumes"
Sep 18 15:55:15.801351 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.801314    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:55:15.805025 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.805006    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:55:15.857133 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857104    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ddt4\" (UniqueName: \"kubernetes.io/projected/a59feb5c-d29c-4281-a2ba-afd314a237d0-kube-api-access-9ddt4\") pod \"a59feb5c-d29c-4281-a2ba-afd314a237d0\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") "
Sep 18 15:55:15.857310 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857155    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-tmp-dir\") pod \"a59feb5c-d29c-4281-a2ba-afd314a237d0\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") "
Sep 18 15:55:15.857310 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857189    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-dshm\") pod \"a59feb5c-d29c-4281-a2ba-afd314a237d0\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") "
Sep 18 15:55:15.857310 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857244    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pgrx\" (UniqueName: \"kubernetes.io/projected/5c8e57a6-40c3-4530-83b9-8b4c502347af-kube-api-access-7pgrx\") pod \"5c8e57a6-40c3-4530-83b9-8b4c502347af\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") "
Sep 18 15:55:15.857488 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857311    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-model-cache\") pod \"5c8e57a6-40c3-4530-83b9-8b4c502347af\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") "
Sep 18 15:55:15.857488 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857379    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8e57a6-40c3-4530-83b9-8b4c502347af-tls-certs\") pod \"5c8e57a6-40c3-4530-83b9-8b4c502347af\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") "
Sep 18 15:55:15.857488 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857424    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-home\") pod \"a59feb5c-d29c-4281-a2ba-afd314a237d0\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") "
Sep 18 15:55:15.857488 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857457    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-model-cache\") pod \"a59feb5c-d29c-4281-a2ba-afd314a237d0\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") "
Sep 18 15:55:15.857488 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857488    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-dshm\") pod \"5c8e57a6-40c3-4530-83b9-8b4c502347af\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") "
Sep 18 15:55:15.857736 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857510    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-tmp-dir\") pod \"5c8e57a6-40c3-4530-83b9-8b4c502347af\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") "
Sep 18 15:55:15.857736 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857518    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a59feb5c-d29c-4281-a2ba-afd314a237d0" (UID: "a59feb5c-d29c-4281-a2ba-afd314a237d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:15.857736 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857544    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-home\") pod \"5c8e57a6-40c3-4530-83b9-8b4c502347af\" (UID: \"5c8e57a6-40c3-4530-83b9-8b4c502347af\") "
Sep 18 15:55:15.857736 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857587    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59feb5c-d29c-4281-a2ba-afd314a237d0-tls-certs\") pod \"a59feb5c-d29c-4281-a2ba-afd314a237d0\" (UID: \"a59feb5c-d29c-4281-a2ba-afd314a237d0\") "
Sep 18 15:55:15.857736 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857686    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-home" (OuterVolumeSpecName: "home") pod "a59feb5c-d29c-4281-a2ba-afd314a237d0" (UID: "a59feb5c-d29c-4281-a2ba-afd314a237d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:15.857736 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.857686    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c8e57a6-40c3-4530-83b9-8b4c502347af" (UID: "5c8e57a6-40c3-4530-83b9-8b4c502347af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:15.858085 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.858025    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-tmp-dir\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.858085 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.858053    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-model-cache\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.858085 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.858066    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-home\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.858384 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.858356    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "a59feb5c-d29c-4281-a2ba-afd314a237d0" (UID: "a59feb5c-d29c-4281-a2ba-afd314a237d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:15.858575 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.858552    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-home" (OuterVolumeSpecName: "home") pod "5c8e57a6-40c3-4530-83b9-8b4c502347af" (UID: "5c8e57a6-40c3-4530-83b9-8b4c502347af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:15.858760 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.858738    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c8e57a6-40c3-4530-83b9-8b4c502347af" (UID: "5c8e57a6-40c3-4530-83b9-8b4c502347af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:15.860300 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.860268    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59feb5c-d29c-4281-a2ba-afd314a237d0-kube-api-access-9ddt4" (OuterVolumeSpecName: "kube-api-access-9ddt4") pod "a59feb5c-d29c-4281-a2ba-afd314a237d0" (UID: "a59feb5c-d29c-4281-a2ba-afd314a237d0"). InnerVolumeSpecName "kube-api-access-9ddt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:55:15.861018 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.860990    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c8e57a6-40c3-4530-83b9-8b4c502347af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c8e57a6-40c3-4530-83b9-8b4c502347af" (UID: "5c8e57a6-40c3-4530-83b9-8b4c502347af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:55:15.861298 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.861273    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8e57a6-40c3-4530-83b9-8b4c502347af-kube-api-access-7pgrx" (OuterVolumeSpecName: "kube-api-access-7pgrx") pod "5c8e57a6-40c3-4530-83b9-8b4c502347af" (UID: "5c8e57a6-40c3-4530-83b9-8b4c502347af"). InnerVolumeSpecName "kube-api-access-7pgrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:55:15.861441 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.861408    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-dshm" (OuterVolumeSpecName: "dshm") pod "a59feb5c-d29c-4281-a2ba-afd314a237d0" (UID: "a59feb5c-d29c-4281-a2ba-afd314a237d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:15.862512 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.862486    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59feb5c-d29c-4281-a2ba-afd314a237d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a59feb5c-d29c-4281-a2ba-afd314a237d0" (UID: "a59feb5c-d29c-4281-a2ba-afd314a237d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:55:15.863396 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.863373    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-dshm" (OuterVolumeSpecName: "dshm") pod "5c8e57a6-40c3-4530-83b9-8b4c502347af" (UID: "5c8e57a6-40c3-4530-83b9-8b4c502347af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:15.959470 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.959439    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59feb5c-d29c-4281-a2ba-afd314a237d0-tls-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.959470 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.959467    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ddt4\" (UniqueName: \"kubernetes.io/projected/a59feb5c-d29c-4281-a2ba-afd314a237d0-kube-api-access-9ddt4\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.959470 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.959477    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-dshm\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.959711 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.959488    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pgrx\" (UniqueName: \"kubernetes.io/projected/5c8e57a6-40c3-4530-83b9-8b4c502347af-kube-api-access-7pgrx\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.959711 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.959497    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8e57a6-40c3-4530-83b9-8b4c502347af-tls-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.959711 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.959505    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a59feb5c-d29c-4281-a2ba-afd314a237d0-model-cache\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.959711 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.959514    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-dshm\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.959711 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.959525    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-tmp-dir\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.959711 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:15.959536    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8e57a6-40c3-4530-83b9-8b4c502347af-home\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:16.584895 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.584836    2559 generic.go:356] "Generic (PLEG): container finished" podID="5c8e57a6-40c3-4530-83b9-8b4c502347af" containerID="f36d8dbf7c9131657d9923ba66c1cf5e3799bb1b0301be0761e228767c6e1d4a" exitCode=0
Sep 18 15:55:16.585396 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.584902    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"
Sep 18 15:55:16.585396 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.584921    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw" event={"ID":"5c8e57a6-40c3-4530-83b9-8b4c502347af","Type":"ContainerDied","Data":"f36d8dbf7c9131657d9923ba66c1cf5e3799bb1b0301be0761e228767c6e1d4a"}
Sep 18 15:55:16.585396 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.584973    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw" event={"ID":"5c8e57a6-40c3-4530-83b9-8b4c502347af","Type":"ContainerDied","Data":"05c633e8043a0442d43e1f99b1fb0165acaeed7fdff817da670738e0099e9cdd"}
Sep 18 15:55:16.585396 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.584997    2559 scope.go:117] "RemoveContainer" containerID="f36d8dbf7c9131657d9923ba66c1cf5e3799bb1b0301be0761e228767c6e1d4a"
Sep 18 15:55:16.586566 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.586545    2559 generic.go:356] "Generic (PLEG): container finished" podID="a59feb5c-d29c-4281-a2ba-afd314a237d0" containerID="b6288d13088e3a9f6295bbf201636c31625206765305256826ead8b6647908f0" exitCode=0
Sep 18 15:55:16.586634 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.586587    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h" event={"ID":"a59feb5c-d29c-4281-a2ba-afd314a237d0","Type":"ContainerDied","Data":"b6288d13088e3a9f6295bbf201636c31625206765305256826ead8b6647908f0"}
Sep 18 15:55:16.586713 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.586632    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h" event={"ID":"a59feb5c-d29c-4281-a2ba-afd314a237d0","Type":"ContainerDied","Data":"25ca155000c219292b49f8414a9e6720fa1b41c9642b404ab20f818cfec39082"}
Sep 18 15:55:16.586713 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.586593    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"
Sep 18 15:55:16.596486 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.596467    2559 scope.go:117] "RemoveContainer" containerID="f36d8dbf7c9131657d9923ba66c1cf5e3799bb1b0301be0761e228767c6e1d4a"
Sep 18 15:55:16.596811 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:55:16.596784    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36d8dbf7c9131657d9923ba66c1cf5e3799bb1b0301be0761e228767c6e1d4a\": container with ID starting with f36d8dbf7c9131657d9923ba66c1cf5e3799bb1b0301be0761e228767c6e1d4a not found: ID does not exist" containerID="f36d8dbf7c9131657d9923ba66c1cf5e3799bb1b0301be0761e228767c6e1d4a"
Sep 18 15:55:16.596908 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.596821    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36d8dbf7c9131657d9923ba66c1cf5e3799bb1b0301be0761e228767c6e1d4a"} err="failed to get container status \"f36d8dbf7c9131657d9923ba66c1cf5e3799bb1b0301be0761e228767c6e1d4a\": rpc error: code = NotFound desc = could not find container \"f36d8dbf7c9131657d9923ba66c1cf5e3799bb1b0301be0761e228767c6e1d4a\": container with ID starting with f36d8dbf7c9131657d9923ba66c1cf5e3799bb1b0301be0761e228767c6e1d4a not found: ID does not exist"
Sep 18 15:55:16.596908 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.596843    2559 scope.go:117] "RemoveContainer" containerID="b6288d13088e3a9f6295bbf201636c31625206765305256826ead8b6647908f0"
Sep 18 15:55:16.604681 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.604658    2559 scope.go:117] "RemoveContainer" containerID="b6288d13088e3a9f6295bbf201636c31625206765305256826ead8b6647908f0"
Sep 18 15:55:16.604967 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:55:16.604949    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6288d13088e3a9f6295bbf201636c31625206765305256826ead8b6647908f0\": container with ID starting with b6288d13088e3a9f6295bbf201636c31625206765305256826ead8b6647908f0 not found: ID does not exist" containerID="b6288d13088e3a9f6295bbf201636c31625206765305256826ead8b6647908f0"
Sep 18 15:55:16.605030 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.604974    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6288d13088e3a9f6295bbf201636c31625206765305256826ead8b6647908f0"} err="failed to get container status \"b6288d13088e3a9f6295bbf201636c31625206765305256826ead8b6647908f0\": rpc error: code = NotFound desc = could not find container \"b6288d13088e3a9f6295bbf201636c31625206765305256826ead8b6647908f0\": container with ID starting with b6288d13088e3a9f6295bbf201636c31625206765305256826ead8b6647908f0 not found: ID does not exist"
Sep 18 15:55:16.609381 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.609359    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"]
Sep 18 15:55:16.618459 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.618438    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/simulator-w3-9792ff-kserve-576cffb5-gnbvw"]
Sep 18 15:55:16.632166 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.632141    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"]
Sep 18 15:55:16.638165 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.638146    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/distinct-2-w3-9792ff-kserve-69db96c9dd-cjz4h"]
Sep 18 15:55:16.819405 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:16.819274    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:17.253367 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:17.253321    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:17.744765 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:17.744730    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c8e57a6-40c3-4530-83b9-8b4c502347af" path="/var/lib/kubelet/pods/5c8e57a6-40c3-4530-83b9-8b4c502347af/volumes"
Sep 18 15:55:17.745177 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:17.745143    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a59feb5c-d29c-4281-a2ba-afd314a237d0" path="/var/lib/kubelet/pods/a59feb5c-d29c-4281-a2ba-afd314a237d0/volumes"
Sep 18 15:55:18.077948 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:18.077818    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:18.876489 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:18.876434    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:19.818818 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:19.818781    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:20.465411 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:20.465378    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:21.091226 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:21.091185    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:21.192012 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:21.191917    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:21.310966 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:21.310928    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:21.331975 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:21.331938    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"]
Sep 18 15:55:21.332260 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:21.332233    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc" podUID="df52776c-9cbe-417e-ab2a-4164812cb020" containerName="istio-proxy" containerID="cri-o://eb97210ade1a97851dfca4f15912a41cbdd3477659439f5433d02669dfd56e61" gracePeriod=30
Sep 18 15:55:21.405892 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:21.405763    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:21.907843 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:21.907805    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:22.561022 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:22.560991    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"
Sep 18 15:55:22.561209 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:22.561075    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"
Sep 18 15:55:22.775335 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:22.775272    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:22.876276 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:22.876180    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:23.679056 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:23.679020    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:24.091255 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:24.091221    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:55:24.597623 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:24.597582    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:26.592844 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.592817    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:55:26.656310 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.656219    2559 generic.go:356] "Generic (PLEG): container finished" podID="df52776c-9cbe-417e-ab2a-4164812cb020" containerID="eb97210ade1a97851dfca4f15912a41cbdd3477659439f5433d02669dfd56e61" exitCode=0
Sep 18 15:55:26.656310 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.656271    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"
Sep 18 15:55:26.656544 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.656302    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc" event={"ID":"df52776c-9cbe-417e-ab2a-4164812cb020","Type":"ContainerDied","Data":"eb97210ade1a97851dfca4f15912a41cbdd3477659439f5433d02669dfd56e61"}
Sep 18 15:55:26.656544 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.656363    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc" event={"ID":"df52776c-9cbe-417e-ab2a-4164812cb020","Type":"ContainerDied","Data":"a6a47c2a2aac4d87f631c487fdda486a417dcf29cbeb763f081307e025cd41b8"}
Sep 18 15:55:26.656544 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.656394    2559 scope.go:117] "RemoveContainer" containerID="eb97210ade1a97851dfca4f15912a41cbdd3477659439f5433d02669dfd56e61"
Sep 18 15:55:26.658595 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.658578    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-workload-certs\") pod \"df52776c-9cbe-417e-ab2a-4164812cb020\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") "
Sep 18 15:55:26.658692 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.658613    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-istio-data\") pod \"df52776c-9cbe-417e-ab2a-4164812cb020\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") "
Sep 18 15:55:26.658692 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.658670    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-credential-socket\") pod \"df52776c-9cbe-417e-ab2a-4164812cb020\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") "
Sep 18 15:55:26.658808 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.658727    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-istio-envoy\") pod \"df52776c-9cbe-417e-ab2a-4164812cb020\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") "
Sep 18 15:55:26.658808 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.658760    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9g2q\" (UniqueName: \"kubernetes.io/projected/df52776c-9cbe-417e-ab2a-4164812cb020-kube-api-access-c9g2q\") pod \"df52776c-9cbe-417e-ab2a-4164812cb020\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") "
Sep 18 15:55:26.658949 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.658837    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df52776c-9cbe-417e-ab2a-4164812cb020-istio-token\") pod \"df52776c-9cbe-417e-ab2a-4164812cb020\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") "
Sep 18 15:55:26.658949 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.658880    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-workload-socket\") pod \"df52776c-9cbe-417e-ab2a-4164812cb020\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") "
Sep 18 15:55:26.658949 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.658921    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-istio-data" (OuterVolumeSpecName: "istio-data") pod "df52776c-9cbe-417e-ab2a-4164812cb020" (UID: "df52776c-9cbe-417e-ab2a-4164812cb020"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:26.658949 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.658934    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df52776c-9cbe-417e-ab2a-4164812cb020-istiod-ca-cert\") pod \"df52776c-9cbe-417e-ab2a-4164812cb020\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") "
Sep 18 15:55:26.658949 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.658933    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df52776c-9cbe-417e-ab2a-4164812cb020" (UID: "df52776c-9cbe-417e-ab2a-4164812cb020"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:26.659205 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.658963    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df52776c-9cbe-417e-ab2a-4164812cb020-istio-podinfo\") pod \"df52776c-9cbe-417e-ab2a-4164812cb020\" (UID: \"df52776c-9cbe-417e-ab2a-4164812cb020\") "
Sep 18 15:55:26.659205 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.659179    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df52776c-9cbe-417e-ab2a-4164812cb020" (UID: "df52776c-9cbe-417e-ab2a-4164812cb020"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:26.659594 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.659358    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-workload-socket\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:26.659594 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.659380    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-workload-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:26.659594 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.659393    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-istio-data\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:26.659594 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.659564    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df52776c-9cbe-417e-ab2a-4164812cb020" (UID: "df52776c-9cbe-417e-ab2a-4164812cb020"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:26.659953 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.659719    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df52776c-9cbe-417e-ab2a-4164812cb020-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df52776c-9cbe-417e-ab2a-4164812cb020" (UID: "df52776c-9cbe-417e-ab2a-4164812cb020"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:55:26.661602 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.661568    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df52776c-9cbe-417e-ab2a-4164812cb020-istio-token" (OuterVolumeSpecName: "istio-token") pod "df52776c-9cbe-417e-ab2a-4164812cb020" (UID: "df52776c-9cbe-417e-ab2a-4164812cb020"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:55:26.661781 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.661759    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df52776c-9cbe-417e-ab2a-4164812cb020" (UID: "df52776c-9cbe-417e-ab2a-4164812cb020"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:26.662015 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.661983    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df52776c-9cbe-417e-ab2a-4164812cb020-kube-api-access-c9g2q" (OuterVolumeSpecName: "kube-api-access-c9g2q") pod "df52776c-9cbe-417e-ab2a-4164812cb020" (UID: "df52776c-9cbe-417e-ab2a-4164812cb020"). InnerVolumeSpecName "kube-api-access-c9g2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:55:26.662548 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.662531    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df52776c-9cbe-417e-ab2a-4164812cb020-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df52776c-9cbe-417e-ab2a-4164812cb020" (UID: "df52776c-9cbe-417e-ab2a-4164812cb020"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 15:55:26.668228 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.668212    2559 scope.go:117] "RemoveContainer" containerID="0ec2dc228e6a5a223ca1ac2445c9c898c0535db1be7e6b32efe84fd97a7974a4"
Sep 18 15:55:26.685752 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.685726    2559 scope.go:117] "RemoveContainer" containerID="eb97210ade1a97851dfca4f15912a41cbdd3477659439f5433d02669dfd56e61"
Sep 18 15:55:26.686069 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:55:26.686050    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb97210ade1a97851dfca4f15912a41cbdd3477659439f5433d02669dfd56e61\": container with ID starting with eb97210ade1a97851dfca4f15912a41cbdd3477659439f5433d02669dfd56e61 not found: ID does not exist" containerID="eb97210ade1a97851dfca4f15912a41cbdd3477659439f5433d02669dfd56e61"
Sep 18 15:55:26.686146 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.686076    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb97210ade1a97851dfca4f15912a41cbdd3477659439f5433d02669dfd56e61"} err="failed to get container status \"eb97210ade1a97851dfca4f15912a41cbdd3477659439f5433d02669dfd56e61\": rpc error: code = NotFound desc = could not find container \"eb97210ade1a97851dfca4f15912a41cbdd3477659439f5433d02669dfd56e61\": container with ID starting with eb97210ade1a97851dfca4f15912a41cbdd3477659439f5433d02669dfd56e61 not found: ID does not exist"
Sep 18 15:55:26.686146 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.686093    2559 scope.go:117] "RemoveContainer" containerID="0ec2dc228e6a5a223ca1ac2445c9c898c0535db1be7e6b32efe84fd97a7974a4"
Sep 18 15:55:26.686331 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:55:26.686313    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec2dc228e6a5a223ca1ac2445c9c898c0535db1be7e6b32efe84fd97a7974a4\": container with ID starting with 0ec2dc228e6a5a223ca1ac2445c9c898c0535db1be7e6b32efe84fd97a7974a4 not found: ID does not exist" containerID="0ec2dc228e6a5a223ca1ac2445c9c898c0535db1be7e6b32efe84fd97a7974a4"
Sep 18 15:55:26.686384 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.686334    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec2dc228e6a5a223ca1ac2445c9c898c0535db1be7e6b32efe84fd97a7974a4"} err="failed to get container status \"0ec2dc228e6a5a223ca1ac2445c9c898c0535db1be7e6b32efe84fd97a7974a4\": rpc error: code = NotFound desc = could not find container \"0ec2dc228e6a5a223ca1ac2445c9c898c0535db1be7e6b32efe84fd97a7974a4\": container with ID starting with 0ec2dc228e6a5a223ca1ac2445c9c898c0535db1be7e6b32efe84fd97a7974a4 not found: ID does not exist"
Sep 18 15:55:26.760998 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.760965    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-credential-socket\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:26.760998 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.760999    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df52776c-9cbe-417e-ab2a-4164812cb020-istio-envoy\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:26.761158 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.761012    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9g2q\" (UniqueName: \"kubernetes.io/projected/df52776c-9cbe-417e-ab2a-4164812cb020-kube-api-access-c9g2q\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:26.761158 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.761025    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df52776c-9cbe-417e-ab2a-4164812cb020-istio-token\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:26.761158 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.761037    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df52776c-9cbe-417e-ab2a-4164812cb020-istiod-ca-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:26.761158 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.761049    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df52776c-9cbe-417e-ab2a-4164812cb020-istio-podinfo\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:55:26.981354 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.981299    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"]
Sep 18 15:55:26.983238 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:26.983212    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9792ff-openshift-default-7cb58c7848-zkxbc"]
Sep 18 15:55:27.743692 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:27.743651    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df52776c-9cbe-417e-ab2a-4164812cb020" path="/var/lib/kubelet/pods/df52776c-9cbe-417e-ab2a-4164812cb020/volumes"
Sep 18 15:55:30.461896 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:30.461847    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:36.685092 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:36.685055    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:36.782106 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:36.782064    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:38.139466 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:38.139426    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:40.536400 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:40.536371    2559 scope.go:117] "RemoveContainer" containerID="45358292a62d46518c76475d4ac25af22a37af0302cf33c28c4a394c35a7de72"
Sep 18 15:55:40.545203 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:40.545181    2559 scope.go:117] "RemoveContainer" containerID="bfff34b8dc2366dfe30f06e1cba60e32de20a2a511933553ec59e18ba13a1290"
Sep 18 15:55:42.155176 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:42.155131    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:52.737700 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:52.737662    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:55:58.593934 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:55:58.593890    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:56:06.060638 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:06.060595    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:56:10.835220 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:10.835182    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:56:15.539672 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.539638    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"]
Sep 18 15:56:15.540124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.540108    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8e57a6-40c3-4530-83b9-8b4c502347af" containerName="main"
Sep 18 15:56:15.540124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.540119    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8e57a6-40c3-4530-83b9-8b4c502347af" containerName="main"
Sep 18 15:56:15.540201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.540130    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df52776c-9cbe-417e-ab2a-4164812cb020" containerName="istio-proxy"
Sep 18 15:56:15.540201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.540136    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="df52776c-9cbe-417e-ab2a-4164812cb020" containerName="istio-proxy"
Sep 18 15:56:15.540201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.540160    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a59feb5c-d29c-4281-a2ba-afd314a237d0" containerName="main"
Sep 18 15:56:15.540201 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.540165    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a59feb5c-d29c-4281-a2ba-afd314a237d0" containerName="main"
Sep 18 15:56:15.540330 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.540237    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="df52776c-9cbe-417e-ab2a-4164812cb020" containerName="istio-proxy"
Sep 18 15:56:15.540330 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.540245    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a59feb5c-d29c-4281-a2ba-afd314a237d0" containerName="main"
Sep 18 15:56:15.540330 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.540254    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="df52776c-9cbe-417e-ab2a-4164812cb020" containerName="istio-proxy"
Sep 18 15:56:15.540330 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.540259    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c8e57a6-40c3-4530-83b9-8b4c502347af" containerName="main"
Sep 18 15:56:15.543591 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.543561    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.546644 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.546529    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-998130\"/\"embedding-w0-998130-kserve-self-signed-certs\""
Sep 18 15:56:15.553574 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.553550    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"]
Sep 18 15:56:15.575466 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.575418    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:56:15.714713 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.714676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-home\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.714948 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.714734    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2776a7-6749-4381-b75b-439365cdd928-tls-certs\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.714948 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.714800    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-dshm\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.714948 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.714920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-tmp-dir\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.715124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.714955    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-model-cache\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.715124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.715070    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsj2w\" (UniqueName: \"kubernetes.io/projected/8b2776a7-6749-4381-b75b-439365cdd928-kube-api-access-nsj2w\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.816342 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.816261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2776a7-6749-4381-b75b-439365cdd928-tls-certs\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.816342 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.816302    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-dshm\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.816342 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.816323    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-tmp-dir\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.816342 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.816344    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-model-cache\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.816668 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.816392    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsj2w\" (UniqueName: \"kubernetes.io/projected/8b2776a7-6749-4381-b75b-439365cdd928-kube-api-access-nsj2w\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.816668 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.816411    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-home\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.816808 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.816779    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-tmp-dir\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.816909 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.816805    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-home\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.816909 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.816845    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-model-cache\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.818684 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.818666    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-dshm\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.818866 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.818840    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2776a7-6749-4381-b75b-439365cdd928-tls-certs\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.824839 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.824822    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsj2w\" (UniqueName: \"kubernetes.io/projected/8b2776a7-6749-4381-b75b-439365cdd928-kube-api-access-nsj2w\") pod \"embedding-w0-998130-kserve-5fdf89fd6b-zksgm\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") " pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:15.854923 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:15.854895    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:16.135467 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:16.135439    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"]
Sep 18 15:56:16.137012 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:56:16.136984    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b2776a7_6749_4381_b75b_439365cdd928.slice/crio-6b34e3f6c2ace9c2b4011dc1dcfec81d4270744e9d0bfc45645c3f94ece83d02 WatchSource:0}: Error finding container 6b34e3f6c2ace9c2b4011dc1dcfec81d4270744e9d0bfc45645c3f94ece83d02: Status 404 returned error can't find the container with id 6b34e3f6c2ace9c2b4011dc1dcfec81d4270744e9d0bfc45645c3f94ece83d02
Sep 18 15:56:16.931907 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:16.931872    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm" event={"ID":"8b2776a7-6749-4381-b75b-439365cdd928","Type":"ContainerStarted","Data":"360612d653c8d125bc5554d9d78b0a672456f31e2da79b1fd7680b069fceb641"}
Sep 18 15:56:16.932288 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:16.931921    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm" event={"ID":"8b2776a7-6749-4381-b75b-439365cdd928","Type":"ContainerStarted","Data":"6b34e3f6c2ace9c2b4011dc1dcfec81d4270744e9d0bfc45645c3f94ece83d02"}
Sep 18 15:56:16.951287 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:16.951242    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm" podStartSLOduration=1.951225761 podStartE2EDuration="1.951225761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:56:16.94893595 +0000 UTC m=+1299.510168090" watchObservedRunningTime="2026-09-18 15:56:16.951225761 +0000 UTC m=+1299.512457900"
Sep 18 15:56:17.771902 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:17.771862    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:56:25.855246 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:25.855216    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:25.855744 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:25.855276    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:25.867770 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:25.867743    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:25.992626 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:25.992596    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:56:40.472203 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:40.472161    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:56:40.581590 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:40.581555    2559 scope.go:117] "RemoveContainer" containerID="d96051573163b01d4f993af806738cc92a98b8cdbf5d4ff2e12f239acb8456f8"
Sep 18 15:56:59.264479 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:59.264399    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27"]
Sep 18 15:56:59.264891 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:59.264882    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df52776c-9cbe-417e-ab2a-4164812cb020" containerName="istio-proxy"
Sep 18 15:56:59.264941 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:59.264896    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="df52776c-9cbe-417e-ab2a-4164812cb020" containerName="istio-proxy"
Sep 18 15:56:59.268099 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:59.268083    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27"
Sep 18 15:56:59.275040 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:59.275021    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 15:56:59.275140 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:59.275123    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 15:56:59.281198 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:59.281175    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27"]
Sep 18 15:56:59.411419 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:59.411386    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84ba007-ce55-4dac-ac6f-0475333d3215-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27\" (UID: \"b84ba007-ce55-4dac-ac6f-0475333d3215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27"
Sep 18 15:56:59.512397 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:59.512360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84ba007-ce55-4dac-ac6f-0475333d3215-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27\" (UID: \"b84ba007-ce55-4dac-ac6f-0475333d3215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27"
Sep 18 15:56:59.513028 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:59.513009    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84ba007-ce55-4dac-ac6f-0475333d3215-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27\" (UID: \"b84ba007-ce55-4dac-ac6f-0475333d3215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27"
Sep 18 15:56:59.577748 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:59.577661    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27"
Sep 18 15:56:59.848706 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:56:59.848680    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27"]
Sep 18 15:56:59.850494 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:56:59.850467    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84ba007_ce55_4dac_ac6f_0475333d3215.slice/crio-6b6b3eb0ccae00b36cb5b0d0b95c45ea738b4c1c8e3f836d49a6ee25fa17979b WatchSource:0}: Error finding container 6b6b3eb0ccae00b36cb5b0d0b95c45ea738b4c1c8e3f836d49a6ee25fa17979b: Status 404 returned error can't find the container with id 6b6b3eb0ccae00b36cb5b0d0b95c45ea738b4c1c8e3f836d49a6ee25fa17979b
Sep 18 15:57:00.070217 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:00.070184    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:00.165983 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:00.165951    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27" event={"ID":"b84ba007-ce55-4dac-ac6f-0475333d3215","Type":"ContainerStarted","Data":"6b6b3eb0ccae00b36cb5b0d0b95c45ea738b4c1c8e3f836d49a6ee25fa17979b"}
Sep 18 15:57:00.957359 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:00.957325    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:01.173379 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:01.173346    2559 generic.go:356] "Generic (PLEG): container finished" podID="b84ba007-ce55-4dac-ac6f-0475333d3215" containerID="4ddd8e4e1e7682de330130168868928459a9f999097fc91a68c2efc83df724d5" exitCode=0
Sep 18 15:57:01.173555 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:01.173406    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27" event={"ID":"b84ba007-ce55-4dac-ac6f-0475333d3215","Type":"ContainerDied","Data":"4ddd8e4e1e7682de330130168868928459a9f999097fc91a68c2efc83df724d5"}
Sep 18 15:57:01.621267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:01.621231    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:02.267043 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:02.267010    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:02.324608 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:02.324585    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27"
Sep 18 15:57:02.441443 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:02.441418    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84ba007-ce55-4dac-ac6f-0475333d3215-maas-api-service-ca\") pod \"b84ba007-ce55-4dac-ac6f-0475333d3215\" (UID: \"b84ba007-ce55-4dac-ac6f-0475333d3215\") "
Sep 18 15:57:02.441806 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:02.441784    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84ba007-ce55-4dac-ac6f-0475333d3215-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b84ba007-ce55-4dac-ac6f-0475333d3215" (UID: "b84ba007-ce55-4dac-ac6f-0475333d3215"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:57:02.542159 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:02.542119    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b84ba007-ce55-4dac-ac6f-0475333d3215-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:03.131970 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:03.131932    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:03.186602 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:03.186568    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27" event={"ID":"b84ba007-ce55-4dac-ac6f-0475333d3215","Type":"ContainerDied","Data":"6b6b3eb0ccae00b36cb5b0d0b95c45ea738b4c1c8e3f836d49a6ee25fa17979b"}
Sep 18 15:57:03.186602 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:03.186595    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27"
Sep 18 15:57:03.186808 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:03.186616    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b6b3eb0ccae00b36cb5b0d0b95c45ea738b4c1c8e3f836d49a6ee25fa17979b"
Sep 18 15:57:03.921263 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:03.921214    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:04.791975 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:04.791946    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:10.968080 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:10.968044    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:57:14.840166 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:14.840126    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:14.878369 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:14.878332    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"]
Sep 18 15:57:14.878624 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:14.878597    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj" podUID="a25faaa2-65e2-4255-8966-843f9482c1c6" containerName="payload-processing" containerID="cri-o://3f66f698277e8ed98cf981d8235871d71c1d55a7ef4488925a2412dde4a79385" gracePeriod=30
Sep 18 15:57:14.889163 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:14.889133    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"]
Sep 18 15:57:14.889420 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:14.889374    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z" podUID="1ca50fd5-9d55-48d6-aee1-05cd69881bc7" containerName="payload-pre-processing" containerID="cri-o://3f9b5352016b1db1ee4653e7a416ce73f7d28763e0773437cf9e4eeba9045861" gracePeriod=30
Sep 18 15:57:14.930110 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:14.930075    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:15.004057 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.004004    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:15.088688 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.088658    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27"]
Sep 18 15:57:15.094604 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.094513    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:15.095534 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.095507    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c56bfef6-ztp27"]
Sep 18 15:57:15.158005 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.157983    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"
Sep 18 15:57:15.175912 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.175884    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:15.182318 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.182290    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"
Sep 18 15:57:15.251629 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.251595    2559 generic.go:356] "Generic (PLEG): container finished" podID="a25faaa2-65e2-4255-8966-843f9482c1c6" containerID="3f66f698277e8ed98cf981d8235871d71c1d55a7ef4488925a2412dde4a79385" exitCode=0
Sep 18 15:57:15.251820 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.251629    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj" event={"ID":"a25faaa2-65e2-4255-8966-843f9482c1c6","Type":"ContainerDied","Data":"3f66f698277e8ed98cf981d8235871d71c1d55a7ef4488925a2412dde4a79385"}
Sep 18 15:57:15.251820 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.251653    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"
Sep 18 15:57:15.251820 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.251686    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj" event={"ID":"a25faaa2-65e2-4255-8966-843f9482c1c6","Type":"ContainerDied","Data":"b8aa33af76ea3146e053dbee6a0dc26730b3bf404efbc7e54d23ff11d0ec5e56"}
Sep 18 15:57:15.251820 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.251710    2559 scope.go:117] "RemoveContainer" containerID="3f66f698277e8ed98cf981d8235871d71c1d55a7ef4488925a2412dde4a79385"
Sep 18 15:57:15.252993 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.252975    2559 generic.go:356] "Generic (PLEG): container finished" podID="1ca50fd5-9d55-48d6-aee1-05cd69881bc7" containerID="3f9b5352016b1db1ee4653e7a416ce73f7d28763e0773437cf9e4eeba9045861" exitCode=0
Sep 18 15:57:15.253066 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.253019    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"
Sep 18 15:57:15.253121 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.253061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z" event={"ID":"1ca50fd5-9d55-48d6-aee1-05cd69881bc7","Type":"ContainerDied","Data":"3f9b5352016b1db1ee4653e7a416ce73f7d28763e0773437cf9e4eeba9045861"}
Sep 18 15:57:15.253121 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.253110    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z" event={"ID":"1ca50fd5-9d55-48d6-aee1-05cd69881bc7","Type":"ContainerDied","Data":"d006577799e67f5eab01f26d4920ece9507ea079287c898e622ddbbf89b5099e"}
Sep 18 15:57:15.256032 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.256012    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxmjh\" (UniqueName: \"kubernetes.io/projected/a25faaa2-65e2-4255-8966-843f9482c1c6-kube-api-access-vxmjh\") pod \"a25faaa2-65e2-4255-8966-843f9482c1c6\" (UID: \"a25faaa2-65e2-4255-8966-843f9482c1c6\") "
Sep 18 15:57:15.256117 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.256042    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25faaa2-65e2-4255-8966-843f9482c1c6-plugins-config-volume\") pod \"a25faaa2-65e2-4255-8966-843f9482c1c6\" (UID: \"a25faaa2-65e2-4255-8966-843f9482c1c6\") "
Sep 18 15:57:15.256117 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.256094    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpxdd\" (UniqueName: \"kubernetes.io/projected/1ca50fd5-9d55-48d6-aee1-05cd69881bc7-kube-api-access-tpxdd\") pod \"1ca50fd5-9d55-48d6-aee1-05cd69881bc7\" (UID: \"1ca50fd5-9d55-48d6-aee1-05cd69881bc7\") "
Sep 18 15:57:15.256191 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.256131    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca50fd5-9d55-48d6-aee1-05cd69881bc7-plugins-config-volume\") pod \"1ca50fd5-9d55-48d6-aee1-05cd69881bc7\" (UID: \"1ca50fd5-9d55-48d6-aee1-05cd69881bc7\") "
Sep 18 15:57:15.256511 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.256484    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25faaa2-65e2-4255-8966-843f9482c1c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a25faaa2-65e2-4255-8966-843f9482c1c6" (UID: "a25faaa2-65e2-4255-8966-843f9482c1c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:57:15.256604 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.256533    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca50fd5-9d55-48d6-aee1-05cd69881bc7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ca50fd5-9d55-48d6-aee1-05cd69881bc7" (UID: "1ca50fd5-9d55-48d6-aee1-05cd69881bc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:57:15.258089 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.258069    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25faaa2-65e2-4255-8966-843f9482c1c6-kube-api-access-vxmjh" (OuterVolumeSpecName: "kube-api-access-vxmjh") pod "a25faaa2-65e2-4255-8966-843f9482c1c6" (UID: "a25faaa2-65e2-4255-8966-843f9482c1c6"). InnerVolumeSpecName "kube-api-access-vxmjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:57:15.258672 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.258655    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca50fd5-9d55-48d6-aee1-05cd69881bc7-kube-api-access-tpxdd" (OuterVolumeSpecName: "kube-api-access-tpxdd") pod "1ca50fd5-9d55-48d6-aee1-05cd69881bc7" (UID: "1ca50fd5-9d55-48d6-aee1-05cd69881bc7"). InnerVolumeSpecName "kube-api-access-tpxdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:57:15.261326 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.261308    2559 scope.go:117] "RemoveContainer" containerID="3f66f698277e8ed98cf981d8235871d71c1d55a7ef4488925a2412dde4a79385"
Sep 18 15:57:15.261600 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:57:15.261582    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f66f698277e8ed98cf981d8235871d71c1d55a7ef4488925a2412dde4a79385\": container with ID starting with 3f66f698277e8ed98cf981d8235871d71c1d55a7ef4488925a2412dde4a79385 not found: ID does not exist" containerID="3f66f698277e8ed98cf981d8235871d71c1d55a7ef4488925a2412dde4a79385"
Sep 18 15:57:15.261680 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.261607    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f66f698277e8ed98cf981d8235871d71c1d55a7ef4488925a2412dde4a79385"} err="failed to get container status \"3f66f698277e8ed98cf981d8235871d71c1d55a7ef4488925a2412dde4a79385\": rpc error: code = NotFound desc = could not find container \"3f66f698277e8ed98cf981d8235871d71c1d55a7ef4488925a2412dde4a79385\": container with ID starting with 3f66f698277e8ed98cf981d8235871d71c1d55a7ef4488925a2412dde4a79385 not found: ID does not exist"
Sep 18 15:57:15.261680 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.261623    2559 scope.go:117] "RemoveContainer" containerID="3f9b5352016b1db1ee4653e7a416ce73f7d28763e0773437cf9e4eeba9045861"
Sep 18 15:57:15.270570 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.270547    2559 scope.go:117] "RemoveContainer" containerID="3f9b5352016b1db1ee4653e7a416ce73f7d28763e0773437cf9e4eeba9045861"
Sep 18 15:57:15.270848 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:57:15.270829    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9b5352016b1db1ee4653e7a416ce73f7d28763e0773437cf9e4eeba9045861\": container with ID starting with 3f9b5352016b1db1ee4653e7a416ce73f7d28763e0773437cf9e4eeba9045861 not found: ID does not exist" containerID="3f9b5352016b1db1ee4653e7a416ce73f7d28763e0773437cf9e4eeba9045861"
Sep 18 15:57:15.270930 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.270869    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9b5352016b1db1ee4653e7a416ce73f7d28763e0773437cf9e4eeba9045861"} err="failed to get container status \"3f9b5352016b1db1ee4653e7a416ce73f7d28763e0773437cf9e4eeba9045861\": rpc error: code = NotFound desc = could not find container \"3f9b5352016b1db1ee4653e7a416ce73f7d28763e0773437cf9e4eeba9045861\": container with ID starting with 3f9b5352016b1db1ee4653e7a416ce73f7d28763e0773437cf9e4eeba9045861 not found: ID does not exist"
Sep 18 15:57:15.357425 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.357341    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxmjh\" (UniqueName: \"kubernetes.io/projected/a25faaa2-65e2-4255-8966-843f9482c1c6-kube-api-access-vxmjh\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:15.357425 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.357368    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25faaa2-65e2-4255-8966-843f9482c1c6-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:15.357425 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.357383    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpxdd\" (UniqueName: \"kubernetes.io/projected/1ca50fd5-9d55-48d6-aee1-05cd69881bc7-kube-api-access-tpxdd\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:15.357425 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.357394    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca50fd5-9d55-48d6-aee1-05cd69881bc7-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:15.509879 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.509827    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:15.581641 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.581610    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"]
Sep 18 15:57:15.589622 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.589592    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c56bfef6-8bc99b845-blhdj"]
Sep 18 15:57:15.614918 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.614818    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"]
Sep 18 15:57:15.618148 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.618125    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c56bfef6-7b7f4f69d6-7xh9z"]
Sep 18 15:57:15.744490 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.744446    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ca50fd5-9d55-48d6-aee1-05cd69881bc7" path="/var/lib/kubelet/pods/1ca50fd5-9d55-48d6-aee1-05cd69881bc7/volumes"
Sep 18 15:57:15.744928 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.744908    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25faaa2-65e2-4255-8966-843f9482c1c6" path="/var/lib/kubelet/pods/a25faaa2-65e2-4255-8966-843f9482c1c6/volumes"
Sep 18 15:57:15.745267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:15.745250    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b84ba007-ce55-4dac-ac6f-0475333d3215" path="/var/lib/kubelet/pods/b84ba007-ce55-4dac-ac6f-0475333d3215/volumes"
Sep 18 15:57:16.484152 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:16.484104    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:17.249725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:17.249684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:17.965538 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:17.965502    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:18.147712 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:18.147664    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:18.814156 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:18.814109    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:19.605049 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:19.605015    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:23.228496 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:23.228461    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:34.141578 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.141537    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:34.269901 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.269868    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs"]
Sep 18 15:57:34.270297 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.270283    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca50fd5-9d55-48d6-aee1-05cd69881bc7" containerName="payload-pre-processing"
Sep 18 15:57:34.270297 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.270297    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca50fd5-9d55-48d6-aee1-05cd69881bc7" containerName="payload-pre-processing"
Sep 18 15:57:34.270429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.270306    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b84ba007-ce55-4dac-ac6f-0475333d3215" containerName="revoke-keys"
Sep 18 15:57:34.270429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.270312    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b84ba007-ce55-4dac-ac6f-0475333d3215" containerName="revoke-keys"
Sep 18 15:57:34.270429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.270327    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25faaa2-65e2-4255-8966-843f9482c1c6" containerName="payload-processing"
Sep 18 15:57:34.270429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.270332    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25faaa2-65e2-4255-8966-843f9482c1c6" containerName="payload-processing"
Sep 18 15:57:34.270429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.270410    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca50fd5-9d55-48d6-aee1-05cd69881bc7" containerName="payload-pre-processing"
Sep 18 15:57:34.270429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.270418    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b84ba007-ce55-4dac-ac6f-0475333d3215" containerName="revoke-keys"
Sep 18 15:57:34.270429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.270424    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25faaa2-65e2-4255-8966-843f9482c1c6" containerName="payload-processing"
Sep 18 15:57:34.273654 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.273634    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs"
Sep 18 15:57:34.278026 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.278007    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 15:57:34.278192 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.278043    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 15:57:34.287370 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.287347    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs"]
Sep 18 15:57:34.331463 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.331433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4ef0c6-3115-4164-9676-e3b7c8e6282c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs\" (UID: \"4c4ef0c6-3115-4164-9676-e3b7c8e6282c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs"
Sep 18 15:57:34.432689 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.432655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4ef0c6-3115-4164-9676-e3b7c8e6282c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs\" (UID: \"4c4ef0c6-3115-4164-9676-e3b7c8e6282c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs"
Sep 18 15:57:34.433283 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.433262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4ef0c6-3115-4164-9676-e3b7c8e6282c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs\" (UID: \"4c4ef0c6-3115-4164-9676-e3b7c8e6282c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs"
Sep 18 15:57:34.502489 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.502451    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:34.584059 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.584027    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs"
Sep 18 15:57:34.852904 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:34.852877    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs"]
Sep 18 15:57:34.854585 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:57:34.854558    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4ef0c6_3115_4164_9676_e3b7c8e6282c.slice/crio-5c99e3c62d6e8e2338720fe4c473cc2f118333547a2cf22133b0c37f76ade4b0 WatchSource:0}: Error finding container 5c99e3c62d6e8e2338720fe4c473cc2f118333547a2cf22133b0c37f76ade4b0: Status 404 returned error can't find the container with id 5c99e3c62d6e8e2338720fe4c473cc2f118333547a2cf22133b0c37f76ade4b0
Sep 18 15:57:35.362622 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:35.362585    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs" event={"ID":"4c4ef0c6-3115-4164-9676-e3b7c8e6282c","Type":"ContainerStarted","Data":"efc273030db7c1e370912a4b0f994d3b19be78762d0d4f4f6509b477e977a961"}
Sep 18 15:57:35.363013 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:35.362649    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs" event={"ID":"4c4ef0c6-3115-4164-9676-e3b7c8e6282c","Type":"ContainerStarted","Data":"5c99e3c62d6e8e2338720fe4c473cc2f118333547a2cf22133b0c37f76ade4b0"}
Sep 18 15:57:35.377093 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:35.377069    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:35.382668 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:35.382627    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs" podStartSLOduration=1.38261234 podStartE2EDuration="1.38261234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:57:35.381090944 +0000 UTC m=+1377.942323102" watchObservedRunningTime="2026-09-18 15:57:35.38261234 +0000 UTC m=+1377.943844479"
Sep 18 15:57:36.073638 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:36.073609    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:36.375739 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:36.375644    2559 generic.go:356] "Generic (PLEG): container finished" podID="4c4ef0c6-3115-4164-9676-e3b7c8e6282c" containerID="efc273030db7c1e370912a4b0f994d3b19be78762d0d4f4f6509b477e977a961" exitCode=0
Sep 18 15:57:36.376160 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:36.375729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs" event={"ID":"4c4ef0c6-3115-4164-9676-e3b7c8e6282c","Type":"ContainerDied","Data":"efc273030db7c1e370912a4b0f994d3b19be78762d0d4f4f6509b477e977a961"}
Sep 18 15:57:36.840746 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:36.840708    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:37.522564 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:37.522536    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs"
Sep 18 15:57:37.664062 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:37.663966    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4ef0c6-3115-4164-9676-e3b7c8e6282c-maas-api-service-ca\") pod \"4c4ef0c6-3115-4164-9676-e3b7c8e6282c\" (UID: \"4c4ef0c6-3115-4164-9676-e3b7c8e6282c\") "
Sep 18 15:57:37.664311 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:37.664287    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4ef0c6-3115-4164-9676-e3b7c8e6282c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c4ef0c6-3115-4164-9676-e3b7c8e6282c" (UID: "4c4ef0c6-3115-4164-9676-e3b7c8e6282c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:57:37.685531 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:37.685499    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:37.764873 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:37.764826    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4ef0c6-3115-4164-9676-e3b7c8e6282c-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:37.873910 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:37.873845    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:57:38.388482 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:38.388449    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs"
Sep 18 15:57:38.388482 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:38.388476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs" event={"ID":"4c4ef0c6-3115-4164-9676-e3b7c8e6282c","Type":"ContainerDied","Data":"5c99e3c62d6e8e2338720fe4c473cc2f118333547a2cf22133b0c37f76ade4b0"}
Sep 18 15:57:38.388700 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:38.388522    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c99e3c62d6e8e2338720fe4c473cc2f118333547a2cf22133b0c37f76ade4b0"
Sep 18 15:57:38.607312 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:38.607277    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:39.126094 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:39.126059    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"]
Sep 18 15:57:39.145011 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:39.144985    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"]
Sep 18 15:57:39.269897 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:39.269842    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:39.607585 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:39.607546    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:40.605372 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:40.605340    2559 scope.go:117] "RemoveContainer" containerID="f5167823de44819f829ccc24c060118427debd3d3d425fd4d801ab49eee6c570"
Sep 18 15:57:44.174119 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:44.174081    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:50.037584 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.037544    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:50.073920 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.073885    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"]
Sep 18 15:57:50.074196 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.074139    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk" podUID="6573b858-2906-4a09-a4e3-55ea7238dbb3" containerName="payload-pre-processing" containerID="cri-o://da4dfe76870bdd3a41ab6f2c61fedf6247337b7edbffa0eb005a5b3d30757912" gracePeriod=30
Sep 18 15:57:50.139800 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.139761    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:50.239833 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.239799    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:50.266468 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.266436    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs"]
Sep 18 15:57:50.269032 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.269004    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-998130-w6lcs"]
Sep 18 15:57:50.328631 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.328608    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"
Sep 18 15:57:50.339742 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.339686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:50.454976 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.454944    2559 generic.go:356] "Generic (PLEG): container finished" podID="6573b858-2906-4a09-a4e3-55ea7238dbb3" containerID="da4dfe76870bdd3a41ab6f2c61fedf6247337b7edbffa0eb005a5b3d30757912" exitCode=0
Sep 18 15:57:50.455147 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.454991    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"
Sep 18 15:57:50.455147 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.455032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk" event={"ID":"6573b858-2906-4a09-a4e3-55ea7238dbb3","Type":"ContainerDied","Data":"da4dfe76870bdd3a41ab6f2c61fedf6247337b7edbffa0eb005a5b3d30757912"}
Sep 18 15:57:50.455147 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.455087    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk" event={"ID":"6573b858-2906-4a09-a4e3-55ea7238dbb3","Type":"ContainerDied","Data":"86b35f16277720db7905b8d72f2ed1ee45bddad6822fbc1431799932d832701d"}
Sep 18 15:57:50.455147 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.455119    2559 scope.go:117] "RemoveContainer" containerID="da4dfe76870bdd3a41ab6f2c61fedf6247337b7edbffa0eb005a5b3d30757912"
Sep 18 15:57:50.465352 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.465334    2559 scope.go:117] "RemoveContainer" containerID="da4dfe76870bdd3a41ab6f2c61fedf6247337b7edbffa0eb005a5b3d30757912"
Sep 18 15:57:50.465630 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:57:50.465604    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4dfe76870bdd3a41ab6f2c61fedf6247337b7edbffa0eb005a5b3d30757912\": container with ID starting with da4dfe76870bdd3a41ab6f2c61fedf6247337b7edbffa0eb005a5b3d30757912 not found: ID does not exist" containerID="da4dfe76870bdd3a41ab6f2c61fedf6247337b7edbffa0eb005a5b3d30757912"
Sep 18 15:57:50.465685 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.465632    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4dfe76870bdd3a41ab6f2c61fedf6247337b7edbffa0eb005a5b3d30757912"} err="failed to get container status \"da4dfe76870bdd3a41ab6f2c61fedf6247337b7edbffa0eb005a5b3d30757912\": rpc error: code = NotFound desc = could not find container \"da4dfe76870bdd3a41ab6f2c61fedf6247337b7edbffa0eb005a5b3d30757912\": container with ID starting with da4dfe76870bdd3a41ab6f2c61fedf6247337b7edbffa0eb005a5b3d30757912 not found: ID does not exist"
Sep 18 15:57:50.479957 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.479932    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkfwk\" (UniqueName: \"kubernetes.io/projected/6573b858-2906-4a09-a4e3-55ea7238dbb3-kube-api-access-pkfwk\") pod \"6573b858-2906-4a09-a4e3-55ea7238dbb3\" (UID: \"6573b858-2906-4a09-a4e3-55ea7238dbb3\") "
Sep 18 15:57:50.480054 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.480002    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6573b858-2906-4a09-a4e3-55ea7238dbb3-plugins-config-volume\") pod \"6573b858-2906-4a09-a4e3-55ea7238dbb3\" (UID: \"6573b858-2906-4a09-a4e3-55ea7238dbb3\") "
Sep 18 15:57:50.480373 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.480352    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6573b858-2906-4a09-a4e3-55ea7238dbb3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6573b858-2906-4a09-a4e3-55ea7238dbb3" (UID: "6573b858-2906-4a09-a4e3-55ea7238dbb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:57:50.481886 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.481850    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6573b858-2906-4a09-a4e3-55ea7238dbb3-kube-api-access-pkfwk" (OuterVolumeSpecName: "kube-api-access-pkfwk") pod "6573b858-2906-4a09-a4e3-55ea7238dbb3" (UID: "6573b858-2906-4a09-a4e3-55ea7238dbb3"). InnerVolumeSpecName "kube-api-access-pkfwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:57:50.581477 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.581402    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkfwk\" (UniqueName: \"kubernetes.io/projected/6573b858-2906-4a09-a4e3-55ea7238dbb3-kube-api-access-pkfwk\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:50.581477 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.581429    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6573b858-2906-4a09-a4e3-55ea7238dbb3-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:50.619541 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.619513    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:50.782691 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.782659    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"]
Sep 18 15:57:50.786239 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.786209    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:50.787277 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:50.787255    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-998130-b47cb667d-5pmzk"]
Sep 18 15:57:50.788832 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:57:50.788810    2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6573b858_2906_4a09_a4e3_55ea7238dbb3.slice/crio-86b35f16277720db7905b8d72f2ed1ee45bddad6822fbc1431799932d832701d\": RecentStats: unable to find data in memory cache]"
Sep 18 15:57:51.635630 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:51.635590    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:51.744057 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:51.744024    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c4ef0c6-3115-4164-9676-e3b7c8e6282c" path="/var/lib/kubelet/pods/4c4ef0c6-3115-4164-9676-e3b7c8e6282c/volumes"
Sep 18 15:57:51.744407 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:51.744394    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6573b858-2906-4a09-a4e3-55ea7238dbb3" path="/var/lib/kubelet/pods/6573b858-2906-4a09-a4e3-55ea7238dbb3/volumes"
Sep 18 15:57:52.419006 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:52.418972    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:53.428136 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:53.428094    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:54.041662 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.041628    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:54.146594 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.146556    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm" podUID="8b2776a7-6749-4381-b75b-439365cdd928" containerName="main" containerID="cri-o://360612d653c8d125bc5554d9d78b0a672456f31e2da79b1fd7680b069fceb641" gracePeriod=45
Sep 18 15:57:54.165573 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.165530    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr" podUID="a989b04c-6b87-4ae3-a66d-b3841bfcbf5b" containerName="main" containerID="cri-o://d6df472a06dae3cef33f6600e40ec79b2120d4772fb15d675c1078df8bad0bb7" gracePeriod=45
Sep 18 15:57:54.471093 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.471066    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:57:54.476492 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.476468    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:57:54.481017 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.480994    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b2776a7-6749-4381-b75b-439365cdd928" containerID="360612d653c8d125bc5554d9d78b0a672456f31e2da79b1fd7680b069fceb641" exitCode=0
Sep 18 15:57:54.481140 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.481031    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"
Sep 18 15:57:54.481140 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.481079    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm" event={"ID":"8b2776a7-6749-4381-b75b-439365cdd928","Type":"ContainerDied","Data":"360612d653c8d125bc5554d9d78b0a672456f31e2da79b1fd7680b069fceb641"}
Sep 18 15:57:54.481140 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.481128    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm" event={"ID":"8b2776a7-6749-4381-b75b-439365cdd928","Type":"ContainerDied","Data":"6b34e3f6c2ace9c2b4011dc1dcfec81d4270744e9d0bfc45645c3f94ece83d02"}
Sep 18 15:57:54.481314 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.481153    2559 scope.go:117] "RemoveContainer" containerID="360612d653c8d125bc5554d9d78b0a672456f31e2da79b1fd7680b069fceb641"
Sep 18 15:57:54.482578 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.482558    2559 generic.go:356] "Generic (PLEG): container finished" podID="a989b04c-6b87-4ae3-a66d-b3841bfcbf5b" containerID="d6df472a06dae3cef33f6600e40ec79b2120d4772fb15d675c1078df8bad0bb7" exitCode=0
Sep 18 15:57:54.482672 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.482592    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"
Sep 18 15:57:54.482672 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.482648    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr" event={"ID":"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b","Type":"ContainerDied","Data":"d6df472a06dae3cef33f6600e40ec79b2120d4772fb15d675c1078df8bad0bb7"}
Sep 18 15:57:54.482756 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.482688    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr" event={"ID":"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b","Type":"ContainerDied","Data":"1ee4b08870605e6d7bc228d3f895c0afbf36f35d5825981310b9448771fcee7d"}
Sep 18 15:57:54.489879 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.489832    2559 scope.go:117] "RemoveContainer" containerID="360612d653c8d125bc5554d9d78b0a672456f31e2da79b1fd7680b069fceb641"
Sep 18 15:57:54.490192 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:57:54.490175    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"360612d653c8d125bc5554d9d78b0a672456f31e2da79b1fd7680b069fceb641\": container with ID starting with 360612d653c8d125bc5554d9d78b0a672456f31e2da79b1fd7680b069fceb641 not found: ID does not exist" containerID="360612d653c8d125bc5554d9d78b0a672456f31e2da79b1fd7680b069fceb641"
Sep 18 15:57:54.490263 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.490200    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360612d653c8d125bc5554d9d78b0a672456f31e2da79b1fd7680b069fceb641"} err="failed to get container status \"360612d653c8d125bc5554d9d78b0a672456f31e2da79b1fd7680b069fceb641\": rpc error: code = NotFound desc = could not find container \"360612d653c8d125bc5554d9d78b0a672456f31e2da79b1fd7680b069fceb641\": container with ID starting with 360612d653c8d125bc5554d9d78b0a672456f31e2da79b1fd7680b069fceb641 not found: ID does not exist"
Sep 18 15:57:54.490263 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.490217    2559 scope.go:117] "RemoveContainer" containerID="d6df472a06dae3cef33f6600e40ec79b2120d4772fb15d675c1078df8bad0bb7"
Sep 18 15:57:54.499959 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.499938    2559 scope.go:117] "RemoveContainer" containerID="d6df472a06dae3cef33f6600e40ec79b2120d4772fb15d675c1078df8bad0bb7"
Sep 18 15:57:54.500307 ip-10-0-130-223 kubenswrapper[2559]: E0918 15:57:54.500284    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6df472a06dae3cef33f6600e40ec79b2120d4772fb15d675c1078df8bad0bb7\": container with ID starting with d6df472a06dae3cef33f6600e40ec79b2120d4772fb15d675c1078df8bad0bb7 not found: ID does not exist" containerID="d6df472a06dae3cef33f6600e40ec79b2120d4772fb15d675c1078df8bad0bb7"
Sep 18 15:57:54.500385 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.500315    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6df472a06dae3cef33f6600e40ec79b2120d4772fb15d675c1078df8bad0bb7"} err="failed to get container status \"d6df472a06dae3cef33f6600e40ec79b2120d4772fb15d675c1078df8bad0bb7\": rpc error: code = NotFound desc = could not find container \"d6df472a06dae3cef33f6600e40ec79b2120d4772fb15d675c1078df8bad0bb7\": container with ID starting with d6df472a06dae3cef33f6600e40ec79b2120d4772fb15d675c1078df8bad0bb7 not found: ID does not exist"
Sep 18 15:57:54.519787 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.519751    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsj2w\" (UniqueName: \"kubernetes.io/projected/8b2776a7-6749-4381-b75b-439365cdd928-kube-api-access-nsj2w\") pod \"8b2776a7-6749-4381-b75b-439365cdd928\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") "
Sep 18 15:57:54.519992 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.519808    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-tls-certs\") pod \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") "
Sep 18 15:57:54.519992 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.519866    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2776a7-6749-4381-b75b-439365cdd928-tls-certs\") pod \"8b2776a7-6749-4381-b75b-439365cdd928\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") "
Sep 18 15:57:54.519992 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.519897    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-home\") pod \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") "
Sep 18 15:57:54.519992 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.519927    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-tmp-dir\") pod \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") "
Sep 18 15:57:54.519992 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.519958    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-model-cache\") pod \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") "
Sep 18 15:57:54.520267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.520019    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-home\") pod \"8b2776a7-6749-4381-b75b-439365cdd928\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") "
Sep 18 15:57:54.520267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.520047    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-dshm\") pod \"8b2776a7-6749-4381-b75b-439365cdd928\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") "
Sep 18 15:57:54.520267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.520094    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-dshm\") pod \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") "
Sep 18 15:57:54.520267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.520168    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-model-cache\") pod \"8b2776a7-6749-4381-b75b-439365cdd928\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") "
Sep 18 15:57:54.520267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.520195    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k42jk\" (UniqueName: \"kubernetes.io/projected/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-kube-api-access-k42jk\") pod \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\" (UID: \"a989b04c-6b87-4ae3-a66d-b3841bfcbf5b\") "
Sep 18 15:57:54.520267 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.520230    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-tmp-dir\") pod \"8b2776a7-6749-4381-b75b-439365cdd928\" (UID: \"8b2776a7-6749-4381-b75b-439365cdd928\") "
Sep 18 15:57:54.520801 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.520772    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a989b04c-6b87-4ae3-a66d-b3841bfcbf5b" (UID: "a989b04c-6b87-4ae3-a66d-b3841bfcbf5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.520923 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.520881    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b2776a7-6749-4381-b75b-439365cdd928" (UID: "8b2776a7-6749-4381-b75b-439365cdd928"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.521078 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.521049    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-home" (OuterVolumeSpecName: "home") pod "8b2776a7-6749-4381-b75b-439365cdd928" (UID: "8b2776a7-6749-4381-b75b-439365cdd928"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.521146 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.521057    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-home" (OuterVolumeSpecName: "home") pod "a989b04c-6b87-4ae3-a66d-b3841bfcbf5b" (UID: "a989b04c-6b87-4ae3-a66d-b3841bfcbf5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.521280 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.521258    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a989b04c-6b87-4ae3-a66d-b3841bfcbf5b" (UID: "a989b04c-6b87-4ae3-a66d-b3841bfcbf5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.521386 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.521355    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b2776a7-6749-4381-b75b-439365cdd928" (UID: "8b2776a7-6749-4381-b75b-439365cdd928"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.525263 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.525234    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-dshm" (OuterVolumeSpecName: "dshm") pod "8b2776a7-6749-4381-b75b-439365cdd928" (UID: "8b2776a7-6749-4381-b75b-439365cdd928"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.525263 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.525234    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2776a7-6749-4381-b75b-439365cdd928-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b2776a7-6749-4381-b75b-439365cdd928" (UID: "8b2776a7-6749-4381-b75b-439365cdd928"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:57:54.525437 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.525339    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a989b04c-6b87-4ae3-a66d-b3841bfcbf5b" (UID: "a989b04c-6b87-4ae3-a66d-b3841bfcbf5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:57:54.526051 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.526011    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2776a7-6749-4381-b75b-439365cdd928-kube-api-access-nsj2w" (OuterVolumeSpecName: "kube-api-access-nsj2w") pod "8b2776a7-6749-4381-b75b-439365cdd928" (UID: "8b2776a7-6749-4381-b75b-439365cdd928"). InnerVolumeSpecName "kube-api-access-nsj2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:57:54.526459 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.526433    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-kube-api-access-k42jk" (OuterVolumeSpecName: "kube-api-access-k42jk") pod "a989b04c-6b87-4ae3-a66d-b3841bfcbf5b" (UID: "a989b04c-6b87-4ae3-a66d-b3841bfcbf5b"). InnerVolumeSpecName "kube-api-access-k42jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:57:54.526757 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.526739    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-dshm" (OuterVolumeSpecName: "dshm") pod "a989b04c-6b87-4ae3-a66d-b3841bfcbf5b" (UID: "a989b04c-6b87-4ae3-a66d-b3841bfcbf5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.621847 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.621750    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-dshm\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.621847 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.621783    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-model-cache\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.621847 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.621797    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k42jk\" (UniqueName: \"kubernetes.io/projected/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-kube-api-access-k42jk\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.621847 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.621809    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-tmp-dir\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.621847 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.621822    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsj2w\" (UniqueName: \"kubernetes.io/projected/8b2776a7-6749-4381-b75b-439365cdd928-kube-api-access-nsj2w\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.621847 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.621832    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-tls-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.621847 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.621843    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2776a7-6749-4381-b75b-439365cdd928-tls-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.622216 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.621870    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-home\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.622216 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.621881    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-tmp-dir\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.622216 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.621891    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b-model-cache\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.622216 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.621900    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-home\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.622216 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.621910    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2776a7-6749-4381-b75b-439365cdd928-dshm\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.806214 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.806181    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"]
Sep 18 15:57:54.816399 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.816375    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-998130/embedding-w0-998130-kserve-5fdf89fd6b-zksgm"]
Sep 18 15:57:54.831643 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.831615    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"]
Sep 18 15:57:54.835251 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:54.835229    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-998130/premium-w0-998130-kserve-6cbc66fc94-hfjhr"]
Sep 18 15:57:55.744903 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:55.744844    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b2776a7-6749-4381-b75b-439365cdd928" path="/var/lib/kubelet/pods/8b2776a7-6749-4381-b75b-439365cdd928/volumes"
Sep 18 15:57:55.745303 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:55.745287    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a989b04c-6b87-4ae3-a66d-b3841bfcbf5b" path="/var/lib/kubelet/pods/a989b04c-6b87-4ae3-a66d-b3841bfcbf5b/volumes"
Sep 18 15:57:56.982469 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:56.982435    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:57.082069 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:57.082031    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:57:57.223681 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:57:57.223649    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:00.191535 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:00.191501    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:00.861533 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:00.861484    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:02.441170 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:02.441131    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:02.527050 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:02.527009    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:06.643220 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:06.643185    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:06.717946 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:06.717912    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:06.917441 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:06.917359    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:07.171411 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:07.171327    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:58:08.238331 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:08.238287    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:08.342523 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:08.342488    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:08.473448 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:08.473421    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:09.206704 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:09.206675    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:09.330886 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:09.330836    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:10.572612 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:10.572580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:16.177352 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:16.177302    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:16.249819 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:16.249787    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:17.431544 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:17.431505    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:17.533844 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:17.533801    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:17.634275 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:17.634232    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:18.179834 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:18.179786    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:24.451313 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:24.451218    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:24.559807 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:24.559774    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:24.651920 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:24.651891    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:25.481353 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:25.481318    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:58:28.896583 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:28.896542    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:30.238534 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:30.238498    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:32.417573 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:32.417538    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:35.149519 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:35.149479    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:35.381237 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:35.381202    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:40.641445 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:40.641413    2559 scope.go:117] "RemoveContainer" containerID="3aa5f518b829a73cef9fd6deaef07035c9114078f17e7946bdf5e7cda3b17a98"
Sep 18 15:58:40.650057 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:40.650026    2559 scope.go:117] "RemoveContainer" containerID="50310bb37208d429af7f649a9c9cf6e9b061b79ed690fd88bf3e4a1a4ef8ef5b"
Sep 18 15:58:40.971347 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:40.971314    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:58:54.562281 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:54.562239    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:55.280883 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:55.280829    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:58:55.641192 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:55.641105    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:55.736568 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:55.736533    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:58:56.312065 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:58:56.312022    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:02.563905 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:02.563844    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:02.648578 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:02.648542    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:02.951169 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:02.951123    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:03.299101 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:03.299017    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:03.643429 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:03.643352    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:05.873564 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:05.873519    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:06.717325 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:06.717286    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:06.815111 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:06.815077    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:07.817141 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:07.817103    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:59:07.870026 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:07.869994    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:08.853725 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:08.853668    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:08.949394 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:08.949363    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:09.506174 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:09.506142    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:10.299522 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.299484    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:10.409411 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.409376    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:10.504646 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.504612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:10.874560 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.874526    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:10.950339 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.950302    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:10.950583 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.950559    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"]
Sep 18 15:59:10.951072 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.951055    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6573b858-2906-4a09-a4e3-55ea7238dbb3" containerName="payload-pre-processing"
Sep 18 15:59:10.951124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.951073    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6573b858-2906-4a09-a4e3-55ea7238dbb3" containerName="payload-pre-processing"
Sep 18 15:59:10.951124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.951082    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a989b04c-6b87-4ae3-a66d-b3841bfcbf5b" containerName="main"
Sep 18 15:59:10.951124 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.951088    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a989b04c-6b87-4ae3-a66d-b3841bfcbf5b" containerName="main"
Sep 18 15:59:10.951224 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.951139    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c4ef0c6-3115-4164-9676-e3b7c8e6282c" containerName="revoke-keys"
Sep 18 15:59:10.951224 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.951145    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c4ef0c6-3115-4164-9676-e3b7c8e6282c" containerName="revoke-keys"
Sep 18 15:59:10.951224 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.951157    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b2776a7-6749-4381-b75b-439365cdd928" containerName="main"
Sep 18 15:59:10.951224 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.951162    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b2776a7-6749-4381-b75b-439365cdd928" containerName="main"
Sep 18 15:59:10.951224 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.951221    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c4ef0c6-3115-4164-9676-e3b7c8e6282c" containerName="revoke-keys"
Sep 18 15:59:10.951390 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.951231    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a989b04c-6b87-4ae3-a66d-b3841bfcbf5b" containerName="main"
Sep 18 15:59:10.951390 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.951237    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b2776a7-6749-4381-b75b-439365cdd928" containerName="main"
Sep 18 15:59:10.951390 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.951244    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6573b858-2906-4a09-a4e3-55ea7238dbb3" containerName="payload-pre-processing"
Sep 18 15:59:10.954105 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.954084    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"
Sep 18 15:59:10.958301 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.958276    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-769e22\""
Sep 18 15:59:10.958591 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.958575    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-769e22-dockercfg-6bp5z\""
Sep 18 15:59:10.970200 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:10.970177    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"]
Sep 18 15:59:11.063871 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:11.063542    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957f52a9-e3bb-49bd-956c-847e912c24e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr\" (UID: \"957f52a9-e3bb-49bd-956c-847e912c24e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"
Sep 18 15:59:11.063871 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:11.063656    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9gc\" (UniqueName: \"kubernetes.io/projected/957f52a9-e3bb-49bd-956c-847e912c24e6-kube-api-access-kf9gc\") pod \"payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr\" (UID: \"957f52a9-e3bb-49bd-956c-847e912c24e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"
Sep 18 15:59:11.165109 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:11.165023    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957f52a9-e3bb-49bd-956c-847e912c24e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr\" (UID: \"957f52a9-e3bb-49bd-956c-847e912c24e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"
Sep 18 15:59:11.165312 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:11.165201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9gc\" (UniqueName: \"kubernetes.io/projected/957f52a9-e3bb-49bd-956c-847e912c24e6-kube-api-access-kf9gc\") pod \"payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr\" (UID: \"957f52a9-e3bb-49bd-956c-847e912c24e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"
Sep 18 15:59:11.165907 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:11.165885    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957f52a9-e3bb-49bd-956c-847e912c24e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr\" (UID: \"957f52a9-e3bb-49bd-956c-847e912c24e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"
Sep 18 15:59:11.174033 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:11.174009    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9gc\" (UniqueName: \"kubernetes.io/projected/957f52a9-e3bb-49bd-956c-847e912c24e6-kube-api-access-kf9gc\") pod \"payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr\" (UID: \"957f52a9-e3bb-49bd-956c-847e912c24e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"
Sep 18 15:59:11.266138 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:11.266105    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"
Sep 18 15:59:11.547007 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:11.546981    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"]
Sep 18 15:59:11.548455 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:59:11.548426    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957f52a9_e3bb_49bd_956c_847e912c24e6.slice/crio-78cf9255dfa073369970b4106bd2ca122e4d481faaa53118ed80d41c66574243 WatchSource:0}: Error finding container 78cf9255dfa073369970b4106bd2ca122e4d481faaa53118ed80d41c66574243: Status 404 returned error can't find the container with id 78cf9255dfa073369970b4106bd2ca122e4d481faaa53118ed80d41c66574243
Sep 18 15:59:11.882602 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:11.882517    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr" event={"ID":"957f52a9-e3bb-49bd-956c-847e912c24e6","Type":"ContainerStarted","Data":"bbfc7681cf915605e092f71ae40b61f3fa8197426fc29b556309554aee8a33c8"}
Sep 18 15:59:11.882602 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:11.882565    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr" event={"ID":"957f52a9-e3bb-49bd-956c-847e912c24e6","Type":"ContainerStarted","Data":"78cf9255dfa073369970b4106bd2ca122e4d481faaa53118ed80d41c66574243"}
Sep 18 15:59:11.882846 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:11.882623    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"
Sep 18 15:59:11.906738 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:11.906687    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr" podStartSLOduration=1.9066734300000001 podStartE2EDuration="1.90667343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:59:11.904766355 +0000 UTC m=+1474.465998497" watchObservedRunningTime="2026-09-18 15:59:11.90667343 +0000 UTC m=+1474.467905615"
Sep 18 15:59:12.092648 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:12.092613    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:12.510949 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:12.510911    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:22.889029 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:22.888997    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"
Sep 18 15:59:24.575804 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:24.575773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:24.675369 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:24.675332    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:27.180113 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:27.180075    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:59:31.411602 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:31.411570    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:32.974691 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:32.974655    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:34.277296 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:34.277259    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:35.334507 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:35.334473    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:36.152412 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:36.152383    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:36.662638 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:36.662603    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:36.979770 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:36.979739    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:37.119607 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:37.119569    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:37.497202 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:37.497170    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:37.811615 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:37.811509    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:37.983372 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:37.983335    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:39.309901 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:39.309838    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:39.965177 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:39.965131    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:40.091897 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:40.091845    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:40.204510 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:40.204478    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:40.684802 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:40.684768    2559 scope.go:117] "RemoveContainer" containerID="fe0ee7e3a9aafb7f17e2d2a39966475f480a1c393a14d94df113c53b122fa02a"
Sep 18 15:59:41.087478 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:41.087392    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:42.165946 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:42.165911    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:43.176773 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:43.176732    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:44.098274 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:44.098241    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:45.034864 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.034805    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:45.141419 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.141384    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:45.631514 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.631478    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:45.730644 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.730597    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:45.830595 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.830561    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"]
Sep 18 15:59:45.835444 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.835429    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"
Sep 18 15:59:45.838922 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.838900    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-dc538f-dockercfg-4vrt4\""
Sep 18 15:59:45.839449 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.839432    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-dc538f\""
Sep 18 15:59:45.848661 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.848634    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"]
Sep 18 15:59:45.918656 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.918587    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"]
Sep 18 15:59:45.922044 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.922024    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"
Sep 18 15:59:45.937516 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.937488    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"]
Sep 18 15:59:45.980272 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.980242    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e050c289-378a-4fed-9085-20872a9cec50-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b\" (UID: \"e050c289-378a-4fed-9085-20872a9cec50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"
Sep 18 15:59:45.980418 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:45.980298    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sndxc\" (UniqueName: \"kubernetes.io/projected/e050c289-378a-4fed-9085-20872a9cec50-kube-api-access-sndxc\") pod \"payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b\" (UID: \"e050c289-378a-4fed-9085-20872a9cec50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"
Sep 18 15:59:46.081627 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.081595    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcbl5\" (UniqueName: \"kubernetes.io/projected/567eaa4a-c671-4b40-be06-04febd230ebc-kube-api-access-rcbl5\") pod \"payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl\" (UID: \"567eaa4a-c671-4b40-be06-04febd230ebc\") " pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"
Sep 18 15:59:46.082022 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.081631    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567eaa4a-c671-4b40-be06-04febd230ebc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl\" (UID: \"567eaa4a-c671-4b40-be06-04febd230ebc\") " pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"
Sep 18 15:59:46.082022 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.081755    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e050c289-378a-4fed-9085-20872a9cec50-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b\" (UID: \"e050c289-378a-4fed-9085-20872a9cec50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"
Sep 18 15:59:46.082022 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.081802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sndxc\" (UniqueName: \"kubernetes.io/projected/e050c289-378a-4fed-9085-20872a9cec50-kube-api-access-sndxc\") pod \"payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b\" (UID: \"e050c289-378a-4fed-9085-20872a9cec50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"
Sep 18 15:59:46.082472 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.082448    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e050c289-378a-4fed-9085-20872a9cec50-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b\" (UID: \"e050c289-378a-4fed-9085-20872a9cec50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"
Sep 18 15:59:46.091161 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.091130    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sndxc\" (UniqueName: \"kubernetes.io/projected/e050c289-378a-4fed-9085-20872a9cec50-kube-api-access-sndxc\") pod \"payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b\" (UID: \"e050c289-378a-4fed-9085-20872a9cec50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"
Sep 18 15:59:46.145371 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.145340    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"
Sep 18 15:59:46.184070 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.184033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcbl5\" (UniqueName: \"kubernetes.io/projected/567eaa4a-c671-4b40-be06-04febd230ebc-kube-api-access-rcbl5\") pod \"payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl\" (UID: \"567eaa4a-c671-4b40-be06-04febd230ebc\") " pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"
Sep 18 15:59:46.184239 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.184086    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567eaa4a-c671-4b40-be06-04febd230ebc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl\" (UID: \"567eaa4a-c671-4b40-be06-04febd230ebc\") " pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"
Sep 18 15:59:46.184873 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.184837    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567eaa4a-c671-4b40-be06-04febd230ebc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl\" (UID: \"567eaa4a-c671-4b40-be06-04febd230ebc\") " pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"
Sep 18 15:59:46.197619 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.197568    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcbl5\" (UniqueName: \"kubernetes.io/projected/567eaa4a-c671-4b40-be06-04febd230ebc-kube-api-access-rcbl5\") pod \"payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl\" (UID: \"567eaa4a-c671-4b40-be06-04febd230ebc\") " pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"
Sep 18 15:59:46.232189 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.232162    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"
Sep 18 15:59:46.424689 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.424665    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"]
Sep 18 15:59:46.426619 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:59:46.426590    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode050c289_378a_4fed_9085_20872a9cec50.slice/crio-1ab7eef30ca15f02bf1700ffae38f4859036e87f17b3676afde610ab87fc7bbd WatchSource:0}: Error finding container 1ab7eef30ca15f02bf1700ffae38f4859036e87f17b3676afde610ab87fc7bbd: Status 404 returned error can't find the container with id 1ab7eef30ca15f02bf1700ffae38f4859036e87f17b3676afde610ab87fc7bbd
Sep 18 15:59:46.522133 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.522099    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"]
Sep 18 15:59:46.525035 ip-10-0-130-223 kubenswrapper[2559]: W0918 15:59:46.525009    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567eaa4a_c671_4b40_be06_04febd230ebc.slice/crio-c07d8c55931867e5069e1520b45d59902a0ce12fbae85c803093f43496b7e736 WatchSource:0}: Error finding container c07d8c55931867e5069e1520b45d59902a0ce12fbae85c803093f43496b7e736: Status 404 returned error can't find the container with id c07d8c55931867e5069e1520b45d59902a0ce12fbae85c803093f43496b7e736
Sep 18 15:59:46.997282 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:46.997245    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:47.069716 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:47.069677    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl" event={"ID":"567eaa4a-c671-4b40-be06-04febd230ebc","Type":"ContainerStarted","Data":"727de51f076016aa3cdb705b1a357a44482d62bcfac6a0c70a5c60f28f340fb0"}
Sep 18 15:59:47.069929 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:47.069748    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl" event={"ID":"567eaa4a-c671-4b40-be06-04febd230ebc","Type":"ContainerStarted","Data":"c07d8c55931867e5069e1520b45d59902a0ce12fbae85c803093f43496b7e736"}
Sep 18 15:59:47.069929 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:47.069781    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"
Sep 18 15:59:47.071497 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:47.071469    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b" event={"ID":"e050c289-378a-4fed-9085-20872a9cec50","Type":"ContainerStarted","Data":"281b4ce39342fa56e4daaf4d1ab676a21a39eab02fd01ff11b07d2ae15495f48"}
Sep 18 15:59:47.071591 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:47.071525    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"
Sep 18 15:59:47.071591 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:47.071548    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b" event={"ID":"e050c289-378a-4fed-9085-20872a9cec50","Type":"ContainerStarted","Data":"1ab7eef30ca15f02bf1700ffae38f4859036e87f17b3676afde610ab87fc7bbd"}
Sep 18 15:59:47.102714 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:47.102670    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl" podStartSLOduration=2.102657373 podStartE2EDuration="2.102657373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:59:47.098876434 +0000 UTC m=+1509.660108571" watchObservedRunningTime="2026-09-18 15:59:47.102657373 +0000 UTC m=+1509.663889513"
Sep 18 15:59:47.131375 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:47.131324    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b" podStartSLOduration=2.131308805 podStartE2EDuration="2.131308805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:59:47.127649406 +0000 UTC m=+1509.688881545" watchObservedRunningTime="2026-09-18 15:59:47.131308805 +0000 UTC m=+1509.692540943"
Sep 18 15:59:47.992677 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:47.992644    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 15:59:50.277782 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:50.277743    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 15:59:58.080378 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:58.080303    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"
Sep 18 15:59:58.080378 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:58.080370    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"
Sep 18 15:59:59.170971 ip-10-0-130-223 kubenswrapper[2559]: I0918 15:59:59.170937    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:00.167024 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.166985    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4"]
Sep 18 16:00:00.170676 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.170638    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4"
Sep 18 16:00:00.172333 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.172306    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj"]
Sep 18 16:00:00.176536 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.176516    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74"]
Sep 18 16:00:00.176646 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.176624    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj"
Sep 18 16:00:00.178511 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.178496    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 16:00:00.180064 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.180007    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74"
Sep 18 16:00:00.183557 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.183540    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4"
Sep 18 16:00:00.193623 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.193599    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj"
Sep 18 16:00:00.197328 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.197292    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4"]
Sep 18 16:00:00.202236 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.201748    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829120-lgq4q"]
Sep 18 16:00:00.210263 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.210239    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74"]
Sep 18 16:00:00.210428 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.210416    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj"]
Sep 18 16:00:00.210539 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.210354    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829120-lgq4q"
Sep 18 16:00:00.231320 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.231280    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829120-lgq4q"]
Sep 18 16:00:00.241512 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.241481    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74"
Sep 18 16:00:00.254154 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.254125    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829120-lgq4q"
Sep 18 16:00:00.491744 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.491710    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4"]
Sep 18 16:00:00.492901 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:00:00.492876    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec5a000_051e_47b7_ad97_d589cd533fe5.slice/crio-7dbefb214d6e3fa2d5e3def7d0eb0ca21c4e54cff0cebeb159ce83c8afb5786e WatchSource:0}: Error finding container 7dbefb214d6e3fa2d5e3def7d0eb0ca21c4e54cff0cebeb159ce83c8afb5786e: Status 404 returned error can't find the container with id 7dbefb214d6e3fa2d5e3def7d0eb0ca21c4e54cff0cebeb159ce83c8afb5786e
Sep 18 16:00:00.494417 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:00:00.494384    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff73ef8_4a05_4222_9133_f388bffff7f9.slice/crio-69a6e6af8f0763df38cf437d981b7433e44a0a2fecb5eb0b21d52cd6c03c499a WatchSource:0}: Error finding container 69a6e6af8f0763df38cf437d981b7433e44a0a2fecb5eb0b21d52cd6c03c499a: Status 404 returned error can't find the container with id 69a6e6af8f0763df38cf437d981b7433e44a0a2fecb5eb0b21d52cd6c03c499a
Sep 18 16:00:00.494557 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.494539    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj"]
Sep 18 16:00:00.560754 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.560406    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829120-lgq4q"]
Sep 18 16:00:00.563766 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:00.563235    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74"]
Sep 18 16:00:00.563766 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:00:00.563341    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd511f8f_6313_4746_8ac2_63c047bfff16.slice/crio-697e9d5be735afaf794f6ceb9869204ea81260c1f8375165e52314b031602c87 WatchSource:0}: Error finding container 697e9d5be735afaf794f6ceb9869204ea81260c1f8375165e52314b031602c87: Status 404 returned error can't find the container with id 697e9d5be735afaf794f6ceb9869204ea81260c1f8375165e52314b031602c87
Sep 18 16:00:00.566327 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:00:00.566305    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d79bb4c_9350_4a30_a1d7_5de653425326.slice/crio-d791801c98a60caa6bb47f89ff8ba5c129be1b275419c6e6b7fe114ba9fd922d WatchSource:0}: Error finding container d791801c98a60caa6bb47f89ff8ba5c129be1b275419c6e6b7fe114ba9fd922d: Status 404 returned error can't find the container with id d791801c98a60caa6bb47f89ff8ba5c129be1b275419c6e6b7fe114ba9fd922d
Sep 18 16:00:01.148916 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:01.148797    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829120-lgq4q" event={"ID":"bd511f8f-6313-4746-8ac2-63c047bfff16","Type":"ContainerStarted","Data":"8dabdbfc7d7feb13fd43d0659b92cf3b3db529d5d6950d9692da3d70ed6a5338"}
Sep 18 16:00:01.148916 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:01.148881    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829120-lgq4q" event={"ID":"bd511f8f-6313-4746-8ac2-63c047bfff16","Type":"ContainerStarted","Data":"697e9d5be735afaf794f6ceb9869204ea81260c1f8375165e52314b031602c87"}
Sep 18 16:00:01.150431 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:01.150407    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74" event={"ID":"0d79bb4c-9350-4a30-a1d7-5de653425326","Type":"ContainerStarted","Data":"26d13209c2d84146dd9feace54a5f5aa789ae0f673b3ce2191ef2829e455ea16"}
Sep 18 16:00:01.150542 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:01.150450    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74" event={"ID":"0d79bb4c-9350-4a30-a1d7-5de653425326","Type":"ContainerStarted","Data":"d791801c98a60caa6bb47f89ff8ba5c129be1b275419c6e6b7fe114ba9fd922d"}
Sep 18 16:00:01.151778 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:01.151754    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj" event={"ID":"4ff73ef8-4a05-4222-9133-f388bffff7f9","Type":"ContainerStarted","Data":"8982b22dcdc04f0dd269cec48dca1692ad381f7cc3a39f7e0c7e1b03141861a6"}
Sep 18 16:00:01.151912 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:01.151793    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj" event={"ID":"4ff73ef8-4a05-4222-9133-f388bffff7f9","Type":"ContainerStarted","Data":"69a6e6af8f0763df38cf437d981b7433e44a0a2fecb5eb0b21d52cd6c03c499a"}
Sep 18 16:00:01.153284 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:01.153261    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4" event={"ID":"fec5a000-051e-47b7-ad97-d589cd533fe5","Type":"ContainerStarted","Data":"7bbe913a1659dcf5cdbfd9ad2ace80b4226e05b1c326419bec18a44782d4454c"}
Sep 18 16:00:01.153377 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:01.153301    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4" event={"ID":"fec5a000-051e-47b7-ad97-d589cd533fe5","Type":"ContainerStarted","Data":"7dbefb214d6e3fa2d5e3def7d0eb0ca21c4e54cff0cebeb159ce83c8afb5786e"}
Sep 18 16:00:01.167547 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:01.167494    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829120-lgq4q" podStartSLOduration=1.167479221 podStartE2EDuration="1.167479221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:00:01.165566891 +0000 UTC m=+1523.726799042" watchObservedRunningTime="2026-09-18 16:00:01.167479221 +0000 UTC m=+1523.728711354"
Sep 18 16:00:01.183460 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:01.183410    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj" podStartSLOduration=1.183395341 podStartE2EDuration="1.183395341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:00:01.181892859 +0000 UTC m=+1523.743125022" watchObservedRunningTime="2026-09-18 16:00:01.183395341 +0000 UTC m=+1523.744627482"
Sep 18 16:00:01.200009 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:01.199958    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74" podStartSLOduration=1.199946309 podStartE2EDuration="1.199946309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:00:01.197920857 +0000 UTC m=+1523.759153016" watchObservedRunningTime="2026-09-18 16:00:01.199946309 +0000 UTC m=+1523.761178441"
Sep 18 16:00:01.212981 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:01.212939    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4" podStartSLOduration=1.212926683 podStartE2EDuration="1.212926683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:00:01.211764214 +0000 UTC m=+1523.772996354" watchObservedRunningTime="2026-09-18 16:00:01.212926683 +0000 UTC m=+1523.774158822"
Sep 18 16:00:02.160086 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:02.160052    2559 generic.go:356] "Generic (PLEG): container finished" podID="bd511f8f-6313-4746-8ac2-63c047bfff16" containerID="8dabdbfc7d7feb13fd43d0659b92cf3b3db529d5d6950d9692da3d70ed6a5338" exitCode=0
Sep 18 16:00:02.160270 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:02.160131    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829120-lgq4q" event={"ID":"bd511f8f-6313-4746-8ac2-63c047bfff16","Type":"ContainerDied","Data":"8dabdbfc7d7feb13fd43d0659b92cf3b3db529d5d6950d9692da3d70ed6a5338"}
Sep 18 16:00:02.161711 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:02.161687    2559 generic.go:356] "Generic (PLEG): container finished" podID="0d79bb4c-9350-4a30-a1d7-5de653425326" containerID="26d13209c2d84146dd9feace54a5f5aa789ae0f673b3ce2191ef2829e455ea16" exitCode=0
Sep 18 16:00:02.161841 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:02.161766    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74" event={"ID":"0d79bb4c-9350-4a30-a1d7-5de653425326","Type":"ContainerDied","Data":"26d13209c2d84146dd9feace54a5f5aa789ae0f673b3ce2191ef2829e455ea16"}
Sep 18 16:00:02.163304 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:02.163281    2559 generic.go:356] "Generic (PLEG): container finished" podID="4ff73ef8-4a05-4222-9133-f388bffff7f9" containerID="8982b22dcdc04f0dd269cec48dca1692ad381f7cc3a39f7e0c7e1b03141861a6" exitCode=0
Sep 18 16:00:02.163399 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:02.163356    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj" event={"ID":"4ff73ef8-4a05-4222-9133-f388bffff7f9","Type":"ContainerDied","Data":"8982b22dcdc04f0dd269cec48dca1692ad381f7cc3a39f7e0c7e1b03141861a6"}
Sep 18 16:00:02.164734 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:02.164714    2559 generic.go:356] "Generic (PLEG): container finished" podID="fec5a000-051e-47b7-ad97-d589cd533fe5" containerID="7bbe913a1659dcf5cdbfd9ad2ace80b4226e05b1c326419bec18a44782d4454c" exitCode=0
Sep 18 16:00:02.164818 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:02.164779    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4" event={"ID":"fec5a000-051e-47b7-ad97-d589cd533fe5","Type":"ContainerDied","Data":"7bbe913a1659dcf5cdbfd9ad2ace80b4226e05b1c326419bec18a44782d4454c"}
Sep 18 16:00:03.312671 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:03.312649    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4"
Sep 18 16:00:03.350811 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:03.350785    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829120-lgq4q"
Sep 18 16:00:03.411630 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:03.411608    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj"
Sep 18 16:00:03.414778 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:03.414757    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74"
Sep 18 16:00:03.784789 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:00:03.784760    2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd511f8f_6313_4746_8ac2_63c047bfff16.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff73ef8_4a05_4222_9133_f388bffff7f9.slice\": RecentStats: unable to find data in memory cache]"
Sep 18 16:00:04.176391 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:04.176313    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829120-lgq4q"
Sep 18 16:00:04.176546 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:04.176313    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829120-lgq4q" event={"ID":"bd511f8f-6313-4746-8ac2-63c047bfff16","Type":"ContainerDied","Data":"697e9d5be735afaf794f6ceb9869204ea81260c1f8375165e52314b031602c87"}
Sep 18 16:00:04.176546 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:04.176468    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697e9d5be735afaf794f6ceb9869204ea81260c1f8375165e52314b031602c87"
Sep 18 16:00:04.177616 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:04.177598    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74"
Sep 18 16:00:04.177727 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:04.177643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74" event={"ID":"0d79bb4c-9350-4a30-a1d7-5de653425326","Type":"ContainerDied","Data":"d791801c98a60caa6bb47f89ff8ba5c129be1b275419c6e6b7fe114ba9fd922d"}
Sep 18 16:00:04.177727 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:04.177680    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d791801c98a60caa6bb47f89ff8ba5c129be1b275419c6e6b7fe114ba9fd922d"
Sep 18 16:00:04.179089 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:04.179061    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj"
Sep 18 16:00:04.179089 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:04.179083    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj" event={"ID":"4ff73ef8-4a05-4222-9133-f388bffff7f9","Type":"ContainerDied","Data":"69a6e6af8f0763df38cf437d981b7433e44a0a2fecb5eb0b21d52cd6c03c499a"}
Sep 18 16:00:04.179263 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:04.179117    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69a6e6af8f0763df38cf437d981b7433e44a0a2fecb5eb0b21d52cd6c03c499a"
Sep 18 16:00:04.180548 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:04.180533    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4"
Sep 18 16:00:04.180636 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:04.180547    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4" event={"ID":"fec5a000-051e-47b7-ad97-d589cd533fe5","Type":"ContainerDied","Data":"7dbefb214d6e3fa2d5e3def7d0eb0ca21c4e54cff0cebeb159ce83c8afb5786e"}
Sep 18 16:00:04.180636 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:04.180575    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dbefb214d6e3fa2d5e3def7d0eb0ca21c4e54cff0cebeb159ce83c8afb5786e"
Sep 18 16:00:07.170824 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:07.170789    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:00:09.958610 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:09.958534    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:10.262191 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:10.262094    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:10.891733 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:10.891693    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:11.742496 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:11.742457    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:12.497676 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:12.497629    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:13.329456 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.329424    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:13.501135 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.501097    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58"]
Sep 18 16:00:13.501531 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.501519    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec5a000-051e-47b7-ad97-d589cd533fe5" containerName="cleanup"
Sep 18 16:00:13.501572 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.501534    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec5a000-051e-47b7-ad97-d589cd533fe5" containerName="cleanup"
Sep 18 16:00:13.501572 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.501550    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd511f8f-6313-4746-8ac2-63c047bfff16" containerName="cleanup"
Sep 18 16:00:13.501572 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.501559    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd511f8f-6313-4746-8ac2-63c047bfff16" containerName="cleanup"
Sep 18 16:00:13.501572 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.501570    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d79bb4c-9350-4a30-a1d7-5de653425326" containerName="cleanup"
Sep 18 16:00:13.501713 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.501579    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d79bb4c-9350-4a30-a1d7-5de653425326" containerName="cleanup"
Sep 18 16:00:13.501713 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.501591    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ff73ef8-4a05-4222-9133-f388bffff7f9" containerName="cleanup"
Sep 18 16:00:13.501713 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.501599    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ff73ef8-4a05-4222-9133-f388bffff7f9" containerName="cleanup"
Sep 18 16:00:13.501713 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.501660    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d79bb4c-9350-4a30-a1d7-5de653425326" containerName="cleanup"
Sep 18 16:00:13.501713 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.501669    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd511f8f-6313-4746-8ac2-63c047bfff16" containerName="cleanup"
Sep 18 16:00:13.501713 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.501676    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ff73ef8-4a05-4222-9133-f388bffff7f9" containerName="cleanup"
Sep 18 16:00:13.501713 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.501682    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec5a000-051e-47b7-ad97-d589cd533fe5" containerName="cleanup"
Sep 18 16:00:13.505068 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.505049    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58"
Sep 18 16:00:13.512457 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.512432    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 16:00:13.513905 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.513881    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:00:13.517188 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.517166    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58"]
Sep 18 16:00:13.634554 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.634458    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2561cacb-0c70-4e19-9eaf-aefd83cda79d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-769e22-xzn58\" (UID: \"2561cacb-0c70-4e19-9eaf-aefd83cda79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58"
Sep 18 16:00:13.735545 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.735505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2561cacb-0c70-4e19-9eaf-aefd83cda79d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-769e22-xzn58\" (UID: \"2561cacb-0c70-4e19-9eaf-aefd83cda79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58"
Sep 18 16:00:13.736176 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.736157    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2561cacb-0c70-4e19-9eaf-aefd83cda79d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-769e22-xzn58\" (UID: \"2561cacb-0c70-4e19-9eaf-aefd83cda79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58"
Sep 18 16:00:13.815807 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:13.815775    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58"
Sep 18 16:00:14.088502 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:14.088472    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58"]
Sep 18 16:00:14.090456 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:00:14.090432    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2561cacb_0c70_4e19_9eaf_aefd83cda79d.slice/crio-38167aee952b5e1a06feefb9de2f2e97fba5e12fe035dfcebcbdcdd82a7c4a1b WatchSource:0}: Error finding container 38167aee952b5e1a06feefb9de2f2e97fba5e12fe035dfcebcbdcdd82a7c4a1b: Status 404 returned error can't find the container with id 38167aee952b5e1a06feefb9de2f2e97fba5e12fe035dfcebcbdcdd82a7c4a1b
Sep 18 16:00:14.222136 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:14.222099    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:14.235632 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:14.235608    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58" event={"ID":"2561cacb-0c70-4e19-9eaf-aefd83cda79d","Type":"ContainerStarted","Data":"38167aee952b5e1a06feefb9de2f2e97fba5e12fe035dfcebcbdcdd82a7c4a1b"}
Sep 18 16:00:14.928208 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:14.928166    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:15.244154 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:15.244120    2559 generic.go:356] "Generic (PLEG): container finished" podID="2561cacb-0c70-4e19-9eaf-aefd83cda79d" containerID="1221c1d382e9ee0cb0034faa524204389c427f69264d9f30cbeb672cd3101e2d" exitCode=0
Sep 18 16:00:15.244328 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:15.244171    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58" event={"ID":"2561cacb-0c70-4e19-9eaf-aefd83cda79d","Type":"ContainerDied","Data":"1221c1d382e9ee0cb0034faa524204389c427f69264d9f30cbeb672cd3101e2d"}
Sep 18 16:00:15.794466 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:15.794433    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:16.378031 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:16.378009    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58"
Sep 18 16:00:16.459973 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:16.459944    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2561cacb-0c70-4e19-9eaf-aefd83cda79d-maas-api-service-ca\") pod \"2561cacb-0c70-4e19-9eaf-aefd83cda79d\" (UID: \"2561cacb-0c70-4e19-9eaf-aefd83cda79d\") "
Sep 18 16:00:16.460351 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:16.460331    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2561cacb-0c70-4e19-9eaf-aefd83cda79d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2561cacb-0c70-4e19-9eaf-aefd83cda79d" (UID: "2561cacb-0c70-4e19-9eaf-aefd83cda79d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:00:16.561294 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:16.561265    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2561cacb-0c70-4e19-9eaf-aefd83cda79d-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:00:16.609759 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:16.609729    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:17.255747 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:17.255712    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58" event={"ID":"2561cacb-0c70-4e19-9eaf-aefd83cda79d","Type":"ContainerDied","Data":"38167aee952b5e1a06feefb9de2f2e97fba5e12fe035dfcebcbdcdd82a7c4a1b"}
Sep 18 16:00:17.255747 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:17.255742    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58"
Sep 18 16:00:17.255969 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:17.255789    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38167aee952b5e1a06feefb9de2f2e97fba5e12fe035dfcebcbdcdd82a7c4a1b"
Sep 18 16:00:17.463250 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:17.463204    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:19.346668 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:19.346631    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:29.123809 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.123763    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:29.152191 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.152163    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"]
Sep 18 16:00:29.152399 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.152373    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr" podUID="957f52a9-e3bb-49bd-956c-847e912c24e6" containerName="payload-processing" containerID="cri-o://bbfc7681cf915605e092f71ae40b61f3fa8197426fc29b556309554aee8a33c8" gracePeriod=30
Sep 18 16:00:29.224136 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.224090    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:29.323168 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.323139    2559 generic.go:356] "Generic (PLEG): container finished" podID="957f52a9-e3bb-49bd-956c-847e912c24e6" containerID="bbfc7681cf915605e092f71ae40b61f3fa8197426fc29b556309554aee8a33c8" exitCode=0
Sep 18 16:00:29.323307 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.323180    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr" event={"ID":"957f52a9-e3bb-49bd-956c-847e912c24e6","Type":"ContainerDied","Data":"bbfc7681cf915605e092f71ae40b61f3fa8197426fc29b556309554aee8a33c8"}
Sep 18 16:00:29.325005 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.324982    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:29.387368 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.387294    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58"]
Sep 18 16:00:29.392788 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.392765    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-769e22-xzn58"]
Sep 18 16:00:29.413368 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.413348    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"
Sep 18 16:00:29.573464 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.573432    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf9gc\" (UniqueName: \"kubernetes.io/projected/957f52a9-e3bb-49bd-956c-847e912c24e6-kube-api-access-kf9gc\") pod \"957f52a9-e3bb-49bd-956c-847e912c24e6\" (UID: \"957f52a9-e3bb-49bd-956c-847e912c24e6\") "
Sep 18 16:00:29.573641 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.573525    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957f52a9-e3bb-49bd-956c-847e912c24e6-plugins-config-volume\") pod \"957f52a9-e3bb-49bd-956c-847e912c24e6\" (UID: \"957f52a9-e3bb-49bd-956c-847e912c24e6\") "
Sep 18 16:00:29.573898 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.573848    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957f52a9-e3bb-49bd-956c-847e912c24e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "957f52a9-e3bb-49bd-956c-847e912c24e6" (UID: "957f52a9-e3bb-49bd-956c-847e912c24e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:00:29.575546 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.575525    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957f52a9-e3bb-49bd-956c-847e912c24e6-kube-api-access-kf9gc" (OuterVolumeSpecName: "kube-api-access-kf9gc") pod "957f52a9-e3bb-49bd-956c-847e912c24e6" (UID: "957f52a9-e3bb-49bd-956c-847e912c24e6"). InnerVolumeSpecName "kube-api-access-kf9gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:00:29.649113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.649043    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:29.674148 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.674126    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf9gc\" (UniqueName: \"kubernetes.io/projected/957f52a9-e3bb-49bd-956c-847e912c24e6-kube-api-access-kf9gc\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:00:29.674148 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.674147    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957f52a9-e3bb-49bd-956c-847e912c24e6-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:00:29.749591 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.749561    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2561cacb-0c70-4e19-9eaf-aefd83cda79d" path="/var/lib/kubelet/pods/2561cacb-0c70-4e19-9eaf-aefd83cda79d/volumes"
Sep 18 16:00:29.809475 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.809449    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:29.911650 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:29.911570    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:30.329620 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:30.329595    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"
Sep 18 16:00:30.330171 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:30.329631    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr" event={"ID":"957f52a9-e3bb-49bd-956c-847e912c24e6","Type":"ContainerDied","Data":"78cf9255dfa073369970b4106bd2ca122e4d481faaa53118ed80d41c66574243"}
Sep 18 16:00:30.330171 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:30.329691    2559 scope.go:117] "RemoveContainer" containerID="bbfc7681cf915605e092f71ae40b61f3fa8197426fc29b556309554aee8a33c8"
Sep 18 16:00:30.362540 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:30.362512    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"]
Sep 18 16:00:30.373644 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:30.373620    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-769e22-6dbfb89bf7-fbbhr"]
Sep 18 16:00:30.769613 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:30.769578    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:31.743548 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:31.743514    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="957f52a9-e3bb-49bd-956c-847e912c24e6" path="/var/lib/kubelet/pods/957f52a9-e3bb-49bd-956c-847e912c24e6/volumes"
Sep 18 16:00:31.898285 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:31.898236    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:32.726915 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:32.726879    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:32.792558 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:32.792520    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:00:33.868249 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:33.868219    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:34.682741 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:34.682697    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:36.142632 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.142563    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:36.219544 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.219506    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:36.449509 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.449467    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:36.830502 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.830418    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8"]
Sep 18 16:00:36.830973 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.830951    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2561cacb-0c70-4e19-9eaf-aefd83cda79d" containerName="revoke-keys"
Sep 18 16:00:36.830973 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.830975    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2561cacb-0c70-4e19-9eaf-aefd83cda79d" containerName="revoke-keys"
Sep 18 16:00:36.831127 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.831000    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957f52a9-e3bb-49bd-956c-847e912c24e6" containerName="payload-processing"
Sep 18 16:00:36.831127 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.831009    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="957f52a9-e3bb-49bd-956c-847e912c24e6" containerName="payload-processing"
Sep 18 16:00:36.831127 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.831109    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="957f52a9-e3bb-49bd-956c-847e912c24e6" containerName="payload-processing"
Sep 18 16:00:36.831127 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.831125    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2561cacb-0c70-4e19-9eaf-aefd83cda79d" containerName="revoke-keys"
Sep 18 16:00:36.835660 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.835644    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8"
Sep 18 16:00:36.838527 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.838510    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:00:36.838622 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.838528    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 16:00:36.842984 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.842963    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8"]
Sep 18 16:00:36.936466 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:36.936431    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca73be60-c803-48b2-aab5-6f91583df823-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8\" (UID: \"ca73be60-c803-48b2-aab5-6f91583df823\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8"
Sep 18 16:00:37.037411 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:37.037379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca73be60-c803-48b2-aab5-6f91583df823-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8\" (UID: \"ca73be60-c803-48b2-aab5-6f91583df823\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8"
Sep 18 16:00:37.037979 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:37.037963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca73be60-c803-48b2-aab5-6f91583df823-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8\" (UID: \"ca73be60-c803-48b2-aab5-6f91583df823\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8"
Sep 18 16:00:37.147237 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:37.147174    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8"
Sep 18 16:00:37.151585 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:37.151559    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:37.418707 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:37.418678    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8"]
Sep 18 16:00:37.419720 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:00:37.419697    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca73be60_c803_48b2_aab5_6f91583df823.slice/crio-759ec496f371538a1faece5558c36e00aacac6ea490fe4b74091b9e0a618d23b WatchSource:0}: Error finding container 759ec496f371538a1faece5558c36e00aacac6ea490fe4b74091b9e0a618d23b: Status 404 returned error can't find the container with id 759ec496f371538a1faece5558c36e00aacac6ea490fe4b74091b9e0a618d23b
Sep 18 16:00:38.358809 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:38.358717    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:38.374730 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:38.374696    2559 generic.go:356] "Generic (PLEG): container finished" podID="ca73be60-c803-48b2-aab5-6f91583df823" containerID="12bd710fd93be75a35178ffd889fe9bf09c34d0ec667bac92b90731073f515d3" exitCode=0
Sep 18 16:00:38.374912 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:38.374763    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8" event={"ID":"ca73be60-c803-48b2-aab5-6f91583df823","Type":"ContainerDied","Data":"12bd710fd93be75a35178ffd889fe9bf09c34d0ec667bac92b90731073f515d3"}
Sep 18 16:00:38.374912 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:38.374800    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8" event={"ID":"ca73be60-c803-48b2-aab5-6f91583df823","Type":"ContainerStarted","Data":"759ec496f371538a1faece5558c36e00aacac6ea490fe4b74091b9e0a618d23b"}
Sep 18 16:00:38.687550 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:38.687508    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:39.486244 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:39.486178    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:39.504043 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:39.504018    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8"
Sep 18 16:00:39.559154 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:39.559125    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca73be60-c803-48b2-aab5-6f91583df823-maas-api-service-ca\") pod \"ca73be60-c803-48b2-aab5-6f91583df823\" (UID: \"ca73be60-c803-48b2-aab5-6f91583df823\") "
Sep 18 16:00:39.559538 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:39.559513    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca73be60-c803-48b2-aab5-6f91583df823-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca73be60-c803-48b2-aab5-6f91583df823" (UID: "ca73be60-c803-48b2-aab5-6f91583df823"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:00:39.660225 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:39.660155    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca73be60-c803-48b2-aab5-6f91583df823-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:00:40.350714 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:40.350665    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:40.385517 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:40.385490    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8"
Sep 18 16:00:40.385517 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:40.385502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8" event={"ID":"ca73be60-c803-48b2-aab5-6f91583df823","Type":"ContainerDied","Data":"759ec496f371538a1faece5558c36e00aacac6ea490fe4b74091b9e0a618d23b"}
Sep 18 16:00:40.385727 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:40.385543    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="759ec496f371538a1faece5558c36e00aacac6ea490fe4b74091b9e0a618d23b"
Sep 18 16:00:41.509776 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:41.509739    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:41.898183 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:41.898089    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:52.403183 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.403141    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:52.428687 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.428640    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"]
Sep 18 16:00:52.428887 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.428842    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl" podUID="567eaa4a-c671-4b40-be06-04febd230ebc" containerName="payload-processing" containerID="cri-o://727de51f076016aa3cdb705b1a357a44482d62bcfac6a0c70a5c60f28f340fb0" gracePeriod=30
Sep 18 16:00:52.435057 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.434998    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"]
Sep 18 16:00:52.436232 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.435981    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b" podUID="e050c289-378a-4fed-9085-20872a9cec50" containerName="payload-pre-processing" containerID="cri-o://281b4ce39342fa56e4daaf4d1ab676a21a39eab02fd01ff11b07d2ae15495f48" gracePeriod=30
Sep 18 16:00:52.501694 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.501659    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:52.629028 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.628996    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8"]
Sep 18 16:00:52.632676 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.632649    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc538f-fwxc8"]
Sep 18 16:00:52.708196 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.708169    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"
Sep 18 16:00:52.710812 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.710795    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"
Sep 18 16:00:52.790433 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.790400    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcbl5\" (UniqueName: \"kubernetes.io/projected/567eaa4a-c671-4b40-be06-04febd230ebc-kube-api-access-rcbl5\") pod \"567eaa4a-c671-4b40-be06-04febd230ebc\" (UID: \"567eaa4a-c671-4b40-be06-04febd230ebc\") "
Sep 18 16:00:52.790590 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.790467    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e050c289-378a-4fed-9085-20872a9cec50-plugins-config-volume\") pod \"e050c289-378a-4fed-9085-20872a9cec50\" (UID: \"e050c289-378a-4fed-9085-20872a9cec50\") "
Sep 18 16:00:52.790590 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.790506    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sndxc\" (UniqueName: \"kubernetes.io/projected/e050c289-378a-4fed-9085-20872a9cec50-kube-api-access-sndxc\") pod \"e050c289-378a-4fed-9085-20872a9cec50\" (UID: \"e050c289-378a-4fed-9085-20872a9cec50\") "
Sep 18 16:00:52.790590 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.790535    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567eaa4a-c671-4b40-be06-04febd230ebc-plugins-config-volume\") pod \"567eaa4a-c671-4b40-be06-04febd230ebc\" (UID: \"567eaa4a-c671-4b40-be06-04febd230ebc\") "
Sep 18 16:00:52.790777 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.790753    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e050c289-378a-4fed-9085-20872a9cec50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e050c289-378a-4fed-9085-20872a9cec50" (UID: "e050c289-378a-4fed-9085-20872a9cec50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:00:52.791227 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.790952    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e050c289-378a-4fed-9085-20872a9cec50-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:00:52.791227 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.790955    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567eaa4a-c671-4b40-be06-04febd230ebc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "567eaa4a-c671-4b40-be06-04febd230ebc" (UID: "567eaa4a-c671-4b40-be06-04febd230ebc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:00:52.792638 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.792617    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567eaa4a-c671-4b40-be06-04febd230ebc-kube-api-access-rcbl5" (OuterVolumeSpecName: "kube-api-access-rcbl5") pod "567eaa4a-c671-4b40-be06-04febd230ebc" (UID: "567eaa4a-c671-4b40-be06-04febd230ebc"). InnerVolumeSpecName "kube-api-access-rcbl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:00:52.792723 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.792635    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e050c289-378a-4fed-9085-20872a9cec50-kube-api-access-sndxc" (OuterVolumeSpecName: "kube-api-access-sndxc") pod "e050c289-378a-4fed-9085-20872a9cec50" (UID: "e050c289-378a-4fed-9085-20872a9cec50"). InnerVolumeSpecName "kube-api-access-sndxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:00:52.891628 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.891598    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcbl5\" (UniqueName: \"kubernetes.io/projected/567eaa4a-c671-4b40-be06-04febd230ebc-kube-api-access-rcbl5\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:00:52.891628 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.891624    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sndxc\" (UniqueName: \"kubernetes.io/projected/e050c289-378a-4fed-9085-20872a9cec50-kube-api-access-sndxc\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:00:52.891628 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:52.891632    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567eaa4a-c671-4b40-be06-04febd230ebc-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:00:53.096337 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.096261    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:53.198091 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.198053    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:53.459663 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.459634    2559 generic.go:356] "Generic (PLEG): container finished" podID="567eaa4a-c671-4b40-be06-04febd230ebc" containerID="727de51f076016aa3cdb705b1a357a44482d62bcfac6a0c70a5c60f28f340fb0" exitCode=0
Sep 18 16:00:53.460085 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.459692    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"
Sep 18 16:00:53.460085 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.459729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl" event={"ID":"567eaa4a-c671-4b40-be06-04febd230ebc","Type":"ContainerDied","Data":"727de51f076016aa3cdb705b1a357a44482d62bcfac6a0c70a5c60f28f340fb0"}
Sep 18 16:00:53.460085 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.459779    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl" event={"ID":"567eaa4a-c671-4b40-be06-04febd230ebc","Type":"ContainerDied","Data":"c07d8c55931867e5069e1520b45d59902a0ce12fbae85c803093f43496b7e736"}
Sep 18 16:00:53.460085 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.459803    2559 scope.go:117] "RemoveContainer" containerID="727de51f076016aa3cdb705b1a357a44482d62bcfac6a0c70a5c60f28f340fb0"
Sep 18 16:00:53.461236 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.461213    2559 generic.go:356] "Generic (PLEG): container finished" podID="e050c289-378a-4fed-9085-20872a9cec50" containerID="281b4ce39342fa56e4daaf4d1ab676a21a39eab02fd01ff11b07d2ae15495f48" exitCode=0
Sep 18 16:00:53.461301 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.461256    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"
Sep 18 16:00:53.461374 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.461282    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b" event={"ID":"e050c289-378a-4fed-9085-20872a9cec50","Type":"ContainerDied","Data":"281b4ce39342fa56e4daaf4d1ab676a21a39eab02fd01ff11b07d2ae15495f48"}
Sep 18 16:00:53.461434 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.461406    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b" event={"ID":"e050c289-378a-4fed-9085-20872a9cec50","Type":"ContainerDied","Data":"1ab7eef30ca15f02bf1700ffae38f4859036e87f17b3676afde610ab87fc7bbd"}
Sep 18 16:00:53.468912 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.468894    2559 scope.go:117] "RemoveContainer" containerID="727de51f076016aa3cdb705b1a357a44482d62bcfac6a0c70a5c60f28f340fb0"
Sep 18 16:00:53.469263 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:00:53.469243    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727de51f076016aa3cdb705b1a357a44482d62bcfac6a0c70a5c60f28f340fb0\": container with ID starting with 727de51f076016aa3cdb705b1a357a44482d62bcfac6a0c70a5c60f28f340fb0 not found: ID does not exist" containerID="727de51f076016aa3cdb705b1a357a44482d62bcfac6a0c70a5c60f28f340fb0"
Sep 18 16:00:53.469351 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.469268    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727de51f076016aa3cdb705b1a357a44482d62bcfac6a0c70a5c60f28f340fb0"} err="failed to get container status \"727de51f076016aa3cdb705b1a357a44482d62bcfac6a0c70a5c60f28f340fb0\": rpc error: code = NotFound desc = could not find container \"727de51f076016aa3cdb705b1a357a44482d62bcfac6a0c70a5c60f28f340fb0\": container with ID starting with 727de51f076016aa3cdb705b1a357a44482d62bcfac6a0c70a5c60f28f340fb0 not found: ID does not exist"
Sep 18 16:00:53.469351 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.469284    2559 scope.go:117] "RemoveContainer" containerID="281b4ce39342fa56e4daaf4d1ab676a21a39eab02fd01ff11b07d2ae15495f48"
Sep 18 16:00:53.477735 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.477720    2559 scope.go:117] "RemoveContainer" containerID="281b4ce39342fa56e4daaf4d1ab676a21a39eab02fd01ff11b07d2ae15495f48"
Sep 18 16:00:53.477967 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:00:53.477946    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281b4ce39342fa56e4daaf4d1ab676a21a39eab02fd01ff11b07d2ae15495f48\": container with ID starting with 281b4ce39342fa56e4daaf4d1ab676a21a39eab02fd01ff11b07d2ae15495f48 not found: ID does not exist" containerID="281b4ce39342fa56e4daaf4d1ab676a21a39eab02fd01ff11b07d2ae15495f48"
Sep 18 16:00:53.478061 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.477976    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281b4ce39342fa56e4daaf4d1ab676a21a39eab02fd01ff11b07d2ae15495f48"} err="failed to get container status \"281b4ce39342fa56e4daaf4d1ab676a21a39eab02fd01ff11b07d2ae15495f48\": rpc error: code = NotFound desc = could not find container \"281b4ce39342fa56e4daaf4d1ab676a21a39eab02fd01ff11b07d2ae15495f48\": container with ID starting with 281b4ce39342fa56e4daaf4d1ab676a21a39eab02fd01ff11b07d2ae15495f48 not found: ID does not exist"
Sep 18 16:00:53.490545 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.490522    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"]
Sep 18 16:00:53.496707 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.496685    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dc538f-7d59dc8559-w7ckl"]
Sep 18 16:00:53.508820 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.508794    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"]
Sep 18 16:00:53.512818 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.512798    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dc538f-c4f65d9f8-8q62b"]
Sep 18 16:00:53.745382 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.745306    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="567eaa4a-c671-4b40-be06-04febd230ebc" path="/var/lib/kubelet/pods/567eaa4a-c671-4b40-be06-04febd230ebc/volumes"
Sep 18 16:00:53.745655 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.745641    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca73be60-c803-48b2-aab5-6f91583df823" path="/var/lib/kubelet/pods/ca73be60-c803-48b2-aab5-6f91583df823/volumes"
Sep 18 16:00:53.745968 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.745953    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e050c289-378a-4fed-9085-20872a9cec50" path="/var/lib/kubelet/pods/e050c289-378a-4fed-9085-20872a9cec50/volumes"
Sep 18 16:00:53.898175 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:53.898114    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:54.074532 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:54.074434    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:54.989782 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:54.989753    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:55.115771 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:55.115723    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:55.576210 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:55.576175    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:00:55.880708 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:55.880619    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:56.782297 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:56.782264    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:58.937476 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:58.937442    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:59.024592 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.024543    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:59.251539 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.251455    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:59.597036 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.596963    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r"]
Sep 18 16:00:59.597362 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.597350    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca73be60-c803-48b2-aab5-6f91583df823" containerName="revoke-keys"
Sep 18 16:00:59.597410 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.597364    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca73be60-c803-48b2-aab5-6f91583df823" containerName="revoke-keys"
Sep 18 16:00:59.597410 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.597376    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="567eaa4a-c671-4b40-be06-04febd230ebc" containerName="payload-processing"
Sep 18 16:00:59.597410 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.597382    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="567eaa4a-c671-4b40-be06-04febd230ebc" containerName="payload-processing"
Sep 18 16:00:59.597410 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.597401    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e050c289-378a-4fed-9085-20872a9cec50" containerName="payload-pre-processing"
Sep 18 16:00:59.597410 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.597406    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e050c289-378a-4fed-9085-20872a9cec50" containerName="payload-pre-processing"
Sep 18 16:00:59.597600 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.597464    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="567eaa4a-c671-4b40-be06-04febd230ebc" containerName="payload-processing"
Sep 18 16:00:59.597600 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.597472    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e050c289-378a-4fed-9085-20872a9cec50" containerName="payload-pre-processing"
Sep 18 16:00:59.597600 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.597479    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca73be60-c803-48b2-aab5-6f91583df823" containerName="revoke-keys"
Sep 18 16:00:59.602161 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.602143    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r"
Sep 18 16:00:59.604934 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.604915    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 16:00:59.605014 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.604918    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:00:59.609434 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.609411    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r"]
Sep 18 16:00:59.748423 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.748380    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:00:59.755471 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.755448    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e1f3c9-9e30-48df-be9b-5d2a1d831f12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r\" (UID: \"91e1f3c9-9e30-48df-be9b-5d2a1d831f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r"
Sep 18 16:00:59.857306 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.857210    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e1f3c9-9e30-48df-be9b-5d2a1d831f12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r\" (UID: \"91e1f3c9-9e30-48df-be9b-5d2a1d831f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r"
Sep 18 16:00:59.857771 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.857752    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e1f3c9-9e30-48df-be9b-5d2a1d831f12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r\" (UID: \"91e1f3c9-9e30-48df-be9b-5d2a1d831f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r"
Sep 18 16:00:59.914322 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:00:59.914292    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r"
Sep 18 16:01:00.184360 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:00.184335    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r"]
Sep 18 16:01:00.185707 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:01:00.185675    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e1f3c9_9e30_48df_be9b_5d2a1d831f12.slice/crio-053572af62550db708da01dd19b565f58f8bdfe28d6f88eae2706ac743ba4ed7 WatchSource:0}: Error finding container 053572af62550db708da01dd19b565f58f8bdfe28d6f88eae2706ac743ba4ed7: Status 404 returned error can't find the container with id 053572af62550db708da01dd19b565f58f8bdfe28d6f88eae2706ac743ba4ed7
Sep 18 16:01:00.441412 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:00.441323    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:00.501146 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:00.501114    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r" event={"ID":"91e1f3c9-9e30-48df-be9b-5d2a1d831f12","Type":"ContainerStarted","Data":"cf1936d15adab7952f3645d260bebb6d93502008ac7b63a3a2efe3c1b0298b90"}
Sep 18 16:01:00.501287 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:00.501170    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r" event={"ID":"91e1f3c9-9e30-48df-be9b-5d2a1d831f12","Type":"ContainerStarted","Data":"053572af62550db708da01dd19b565f58f8bdfe28d6f88eae2706ac743ba4ed7"}
Sep 18 16:01:00.530254 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:00.530207    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r" podStartSLOduration=1.530193366 podStartE2EDuration="1.530193366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:01:00.528824506 +0000 UTC m=+1583.090056645" watchObservedRunningTime="2026-09-18 16:01:00.530193366 +0000 UTC m=+1583.091425505"
Sep 18 16:01:01.228542 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:01.228513    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:01.507915 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:01.507830    2559 generic.go:356] "Generic (PLEG): container finished" podID="91e1f3c9-9e30-48df-be9b-5d2a1d831f12" containerID="cf1936d15adab7952f3645d260bebb6d93502008ac7b63a3a2efe3c1b0298b90" exitCode=0
Sep 18 16:01:01.507915 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:01.507884    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r" event={"ID":"91e1f3c9-9e30-48df-be9b-5d2a1d831f12","Type":"ContainerDied","Data":"cf1936d15adab7952f3645d260bebb6d93502008ac7b63a3a2efe3c1b0298b90"}
Sep 18 16:01:01.529973 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:01.529938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:01.993530 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:01.993492    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:02.645092 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:02.645068    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r"
Sep 18 16:01:02.784130 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:02.784044    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e1f3c9-9e30-48df-be9b-5d2a1d831f12-maas-api-service-ca\") pod \"91e1f3c9-9e30-48df-be9b-5d2a1d831f12\" (UID: \"91e1f3c9-9e30-48df-be9b-5d2a1d831f12\") "
Sep 18 16:01:02.784392 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:02.784371    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e1f3c9-9e30-48df-be9b-5d2a1d831f12-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91e1f3c9-9e30-48df-be9b-5d2a1d831f12" (UID: "91e1f3c9-9e30-48df-be9b-5d2a1d831f12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:01:02.885552 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:02.885520    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e1f3c9-9e30-48df-be9b-5d2a1d831f12-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:01:02.966150 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:02.966115    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:03.439681 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:03.439646    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:03.519332 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:03.519304    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r"
Sep 18 16:01:03.519332 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:03.519330    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r" event={"ID":"91e1f3c9-9e30-48df-be9b-5d2a1d831f12","Type":"ContainerDied","Data":"053572af62550db708da01dd19b565f58f8bdfe28d6f88eae2706ac743ba4ed7"}
Sep 18 16:01:03.519546 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:03.519370    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="053572af62550db708da01dd19b565f58f8bdfe28d6f88eae2706ac743ba4ed7"
Sep 18 16:01:10.447951 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.447914    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:10.461765 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.461735    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74"]
Sep 18 16:01:10.470837 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.470811    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-195881-29829120-whq74"]
Sep 18 16:01:10.513954 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.513926    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"]
Sep 18 16:01:10.514237 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.514205    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226" podUID="732529c0-3590-4c3f-b1ae-13a1cdf477b4" containerName="payload-processing" containerID="cri-o://c4cc69f3efefc63516575103f254880454a030357a5a7dc48726f6bc9746c452" gracePeriod=30
Sep 18 16:01:10.527522 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.527474    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"]
Sep 18 16:01:10.527992 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.527964    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9" podUID="58e9df99-8c7d-4055-a1de-59a80d25ea41" containerName="payload-pre-processing" containerID="cri-o://6772eb3c1cfcefd7f5bc334a65f7f9a5ef40ad3282dcc64ed02712dbe85159ab" gracePeriod=30
Sep 18 16:01:10.565396 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.565359    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:10.666503 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.666453    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:10.777558 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.777534    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"
Sep 18 16:01:10.811763 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.811738    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"
Sep 18 16:01:10.897337 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.897304    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r"]
Sep 18 16:01:10.900109 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.900082    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:10.904678 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.904655    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-195881-zb24r"]
Sep 18 16:01:10.958259 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.958233    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-925k5\" (UniqueName: \"kubernetes.io/projected/58e9df99-8c7d-4055-a1de-59a80d25ea41-kube-api-access-925k5\") pod \"58e9df99-8c7d-4055-a1de-59a80d25ea41\" (UID: \"58e9df99-8c7d-4055-a1de-59a80d25ea41\") "
Sep 18 16:01:10.958411 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.958284    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732529c0-3590-4c3f-b1ae-13a1cdf477b4-plugins-config-volume\") pod \"732529c0-3590-4c3f-b1ae-13a1cdf477b4\" (UID: \"732529c0-3590-4c3f-b1ae-13a1cdf477b4\") "
Sep 18 16:01:10.958411 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.958303    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7g4q\" (UniqueName: \"kubernetes.io/projected/732529c0-3590-4c3f-b1ae-13a1cdf477b4-kube-api-access-k7g4q\") pod \"732529c0-3590-4c3f-b1ae-13a1cdf477b4\" (UID: \"732529c0-3590-4c3f-b1ae-13a1cdf477b4\") "
Sep 18 16:01:10.958411 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.958341    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e9df99-8c7d-4055-a1de-59a80d25ea41-plugins-config-volume\") pod \"58e9df99-8c7d-4055-a1de-59a80d25ea41\" (UID: \"58e9df99-8c7d-4055-a1de-59a80d25ea41\") "
Sep 18 16:01:10.958735 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.958709    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732529c0-3590-4c3f-b1ae-13a1cdf477b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "732529c0-3590-4c3f-b1ae-13a1cdf477b4" (UID: "732529c0-3590-4c3f-b1ae-13a1cdf477b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:01:10.958844 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.958747    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e9df99-8c7d-4055-a1de-59a80d25ea41-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58e9df99-8c7d-4055-a1de-59a80d25ea41" (UID: "58e9df99-8c7d-4055-a1de-59a80d25ea41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:01:10.960493 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.960476    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e9df99-8c7d-4055-a1de-59a80d25ea41-kube-api-access-925k5" (OuterVolumeSpecName: "kube-api-access-925k5") pod "58e9df99-8c7d-4055-a1de-59a80d25ea41" (UID: "58e9df99-8c7d-4055-a1de-59a80d25ea41"). InnerVolumeSpecName "kube-api-access-925k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:01:10.960723 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:10.960685    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732529c0-3590-4c3f-b1ae-13a1cdf477b4-kube-api-access-k7g4q" (OuterVolumeSpecName: "kube-api-access-k7g4q") pod "732529c0-3590-4c3f-b1ae-13a1cdf477b4" (UID: "732529c0-3590-4c3f-b1ae-13a1cdf477b4"). InnerVolumeSpecName "kube-api-access-k7g4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:01:11.060145 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.060046    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-925k5\" (UniqueName: \"kubernetes.io/projected/58e9df99-8c7d-4055-a1de-59a80d25ea41-kube-api-access-925k5\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:01:11.060145 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.060083    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732529c0-3590-4c3f-b1ae-13a1cdf477b4-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:01:11.060145 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.060095    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7g4q\" (UniqueName: \"kubernetes.io/projected/732529c0-3590-4c3f-b1ae-13a1cdf477b4-kube-api-access-k7g4q\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:01:11.060145 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.060106    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e9df99-8c7d-4055-a1de-59a80d25ea41-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:01:11.273347 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.273312    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:11.371500 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.371406    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:11.567246 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.567209    2559 generic.go:356] "Generic (PLEG): container finished" podID="58e9df99-8c7d-4055-a1de-59a80d25ea41" containerID="6772eb3c1cfcefd7f5bc334a65f7f9a5ef40ad3282dcc64ed02712dbe85159ab" exitCode=0
Sep 18 16:01:11.567684 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.567264    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"
Sep 18 16:01:11.567684 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.567304    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9" event={"ID":"58e9df99-8c7d-4055-a1de-59a80d25ea41","Type":"ContainerDied","Data":"6772eb3c1cfcefd7f5bc334a65f7f9a5ef40ad3282dcc64ed02712dbe85159ab"}
Sep 18 16:01:11.567684 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.567375    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9" event={"ID":"58e9df99-8c7d-4055-a1de-59a80d25ea41","Type":"ContainerDied","Data":"2b85d7088d213bad167e8dbf89fd8ff53a3d0b0e235dbcec39d1c3d5ce6046bc"}
Sep 18 16:01:11.567684 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.567405    2559 scope.go:117] "RemoveContainer" containerID="6772eb3c1cfcefd7f5bc334a65f7f9a5ef40ad3282dcc64ed02712dbe85159ab"
Sep 18 16:01:11.568744 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.568678    2559 generic.go:356] "Generic (PLEG): container finished" podID="732529c0-3590-4c3f-b1ae-13a1cdf477b4" containerID="c4cc69f3efefc63516575103f254880454a030357a5a7dc48726f6bc9746c452" exitCode=0
Sep 18 16:01:11.568809 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.568743    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"
Sep 18 16:01:11.568809 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.568765    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226" event={"ID":"732529c0-3590-4c3f-b1ae-13a1cdf477b4","Type":"ContainerDied","Data":"c4cc69f3efefc63516575103f254880454a030357a5a7dc48726f6bc9746c452"}
Sep 18 16:01:11.568809 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.568797    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226" event={"ID":"732529c0-3590-4c3f-b1ae-13a1cdf477b4","Type":"ContainerDied","Data":"7d914971de09bbfc8ce8337bee564ccb6ed1b98fe45755022819aa4d596851de"}
Sep 18 16:01:11.576317 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.576302    2559 scope.go:117] "RemoveContainer" containerID="6772eb3c1cfcefd7f5bc334a65f7f9a5ef40ad3282dcc64ed02712dbe85159ab"
Sep 18 16:01:11.576628 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:01:11.576604    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6772eb3c1cfcefd7f5bc334a65f7f9a5ef40ad3282dcc64ed02712dbe85159ab\": container with ID starting with 6772eb3c1cfcefd7f5bc334a65f7f9a5ef40ad3282dcc64ed02712dbe85159ab not found: ID does not exist" containerID="6772eb3c1cfcefd7f5bc334a65f7f9a5ef40ad3282dcc64ed02712dbe85159ab"
Sep 18 16:01:11.576727 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.576633    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6772eb3c1cfcefd7f5bc334a65f7f9a5ef40ad3282dcc64ed02712dbe85159ab"} err="failed to get container status \"6772eb3c1cfcefd7f5bc334a65f7f9a5ef40ad3282dcc64ed02712dbe85159ab\": rpc error: code = NotFound desc = could not find container \"6772eb3c1cfcefd7f5bc334a65f7f9a5ef40ad3282dcc64ed02712dbe85159ab\": container with ID starting with 6772eb3c1cfcefd7f5bc334a65f7f9a5ef40ad3282dcc64ed02712dbe85159ab not found: ID does not exist"
Sep 18 16:01:11.576727 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.576648    2559 scope.go:117] "RemoveContainer" containerID="c4cc69f3efefc63516575103f254880454a030357a5a7dc48726f6bc9746c452"
Sep 18 16:01:11.587415 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.587394    2559 scope.go:117] "RemoveContainer" containerID="c4cc69f3efefc63516575103f254880454a030357a5a7dc48726f6bc9746c452"
Sep 18 16:01:11.587692 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:01:11.587671    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4cc69f3efefc63516575103f254880454a030357a5a7dc48726f6bc9746c452\": container with ID starting with c4cc69f3efefc63516575103f254880454a030357a5a7dc48726f6bc9746c452 not found: ID does not exist" containerID="c4cc69f3efefc63516575103f254880454a030357a5a7dc48726f6bc9746c452"
Sep 18 16:01:11.587770 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.587698    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cc69f3efefc63516575103f254880454a030357a5a7dc48726f6bc9746c452"} err="failed to get container status \"c4cc69f3efefc63516575103f254880454a030357a5a7dc48726f6bc9746c452\": rpc error: code = NotFound desc = could not find container \"c4cc69f3efefc63516575103f254880454a030357a5a7dc48726f6bc9746c452\": container with ID starting with c4cc69f3efefc63516575103f254880454a030357a5a7dc48726f6bc9746c452 not found: ID does not exist"
Sep 18 16:01:11.596235 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.596214    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"]
Sep 18 16:01:11.600499 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.600479    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-195881-6f8744b875-f8226"]
Sep 18 16:01:11.616041 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.616019    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"]
Sep 18 16:01:11.618187 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.618168    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-195881-6596ccd5d-bwbz9"]
Sep 18 16:01:11.749241 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.749207    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d79bb4c-9350-4a30-a1d7-5de653425326" path="/var/lib/kubelet/pods/0d79bb4c-9350-4a30-a1d7-5de653425326/volumes"
Sep 18 16:01:11.749537 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.749515    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58e9df99-8c7d-4055-a1de-59a80d25ea41" path="/var/lib/kubelet/pods/58e9df99-8c7d-4055-a1de-59a80d25ea41/volumes"
Sep 18 16:01:11.749821 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.749810    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="732529c0-3590-4c3f-b1ae-13a1cdf477b4" path="/var/lib/kubelet/pods/732529c0-3590-4c3f-b1ae-13a1cdf477b4/volumes"
Sep 18 16:01:11.750132 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:11.750120    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e1f3c9-9e30-48df-be9b-5d2a1d831f12" path="/var/lib/kubelet/pods/91e1f3c9-9e30-48df-be9b-5d2a1d831f12/volumes"
Sep 18 16:01:12.051176 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:12.051094    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:12.669634 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:12.669597    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:13.562760 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:13.562722    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:15.082708 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:15.082669    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:01:17.436682 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:17.436637    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:17.543587 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:17.543553    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:17.798133 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:17.798010    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:18.136984 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.136905    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9"]
Sep 18 16:01:18.137307 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.137294    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e1f3c9-9e30-48df-be9b-5d2a1d831f12" containerName="revoke-keys"
Sep 18 16:01:18.137349 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.137308    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e1f3c9-9e30-48df-be9b-5d2a1d831f12" containerName="revoke-keys"
Sep 18 16:01:18.137349 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.137321    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732529c0-3590-4c3f-b1ae-13a1cdf477b4" containerName="payload-processing"
Sep 18 16:01:18.137349 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.137326    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="732529c0-3590-4c3f-b1ae-13a1cdf477b4" containerName="payload-processing"
Sep 18 16:01:18.137349 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.137333    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e9df99-8c7d-4055-a1de-59a80d25ea41" containerName="payload-pre-processing"
Sep 18 16:01:18.137349 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.137338    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e9df99-8c7d-4055-a1de-59a80d25ea41" containerName="payload-pre-processing"
Sep 18 16:01:18.137508 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.137394    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e1f3c9-9e30-48df-be9b-5d2a1d831f12" containerName="revoke-keys"
Sep 18 16:01:18.137508 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.137402    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e9df99-8c7d-4055-a1de-59a80d25ea41" containerName="payload-pre-processing"
Sep 18 16:01:18.137508 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.137411    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="732529c0-3590-4c3f-b1ae-13a1cdf477b4" containerName="payload-processing"
Sep 18 16:01:18.141921 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.141894    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9"
Sep 18 16:01:18.153406 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.146840    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:01:18.153406 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.146955    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 16:01:18.163976 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.163944    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9"]
Sep 18 16:01:18.222906 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.222834    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa23d8a-6133-409e-a8e7-41b438d06196-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9\" (UID: \"5aa23d8a-6133-409e-a8e7-41b438d06196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9"
Sep 18 16:01:18.323822 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.323777    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa23d8a-6133-409e-a8e7-41b438d06196-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9\" (UID: \"5aa23d8a-6133-409e-a8e7-41b438d06196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9"
Sep 18 16:01:18.324396 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.324375    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa23d8a-6133-409e-a8e7-41b438d06196-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9\" (UID: \"5aa23d8a-6133-409e-a8e7-41b438d06196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9"
Sep 18 16:01:18.462683 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.462646    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9"
Sep 18 16:01:18.504360 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.504293    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:18.766616 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:18.766532    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9"]
Sep 18 16:01:18.769793 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:01:18.769761    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa23d8a_6133_409e_a8e7_41b438d06196.slice/crio-0f1c03e688b284d90a6e5404bc46dfa936b841812a6f076c26ecfb0fede6b750 WatchSource:0}: Error finding container 0f1c03e688b284d90a6e5404bc46dfa936b841812a6f076c26ecfb0fede6b750: Status 404 returned error can't find the container with id 0f1c03e688b284d90a6e5404bc46dfa936b841812a6f076c26ecfb0fede6b750
Sep 18 16:01:19.614703 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:19.614669    2559 generic.go:356] "Generic (PLEG): container finished" podID="5aa23d8a-6133-409e-a8e7-41b438d06196" containerID="bd441383c96cf5e7ca0de6d1c94a77080ad301a8a71e8c4343c4cbc8128e769c" exitCode=0
Sep 18 16:01:19.615175 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:19.614756    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9" event={"ID":"5aa23d8a-6133-409e-a8e7-41b438d06196","Type":"ContainerDied","Data":"bd441383c96cf5e7ca0de6d1c94a77080ad301a8a71e8c4343c4cbc8128e769c"}
Sep 18 16:01:19.615175 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:19.614810    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9" event={"ID":"5aa23d8a-6133-409e-a8e7-41b438d06196","Type":"ContainerStarted","Data":"0f1c03e688b284d90a6e5404bc46dfa936b841812a6f076c26ecfb0fede6b750"}
Sep 18 16:01:19.659670 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:19.659629    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:20.388695 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:20.388666    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:20.758519 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:20.758490    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9"
Sep 18 16:01:20.842103 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:20.842073    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa23d8a-6133-409e-a8e7-41b438d06196-maas-api-service-ca\") pod \"5aa23d8a-6133-409e-a8e7-41b438d06196\" (UID: \"5aa23d8a-6133-409e-a8e7-41b438d06196\") "
Sep 18 16:01:20.842424 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:20.842400    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa23d8a-6133-409e-a8e7-41b438d06196-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5aa23d8a-6133-409e-a8e7-41b438d06196" (UID: "5aa23d8a-6133-409e-a8e7-41b438d06196"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:01:20.942741 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:20.942709    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa23d8a-6133-409e-a8e7-41b438d06196-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:01:21.158673 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:21.158584    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:21.629168 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:21.629143    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9"
Sep 18 16:01:21.629346 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:21.629142    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9" event={"ID":"5aa23d8a-6133-409e-a8e7-41b438d06196","Type":"ContainerDied","Data":"0f1c03e688b284d90a6e5404bc46dfa936b841812a6f076c26ecfb0fede6b750"}
Sep 18 16:01:21.629346 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:21.629253    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f1c03e688b284d90a6e5404bc46dfa936b841812a6f076c26ecfb0fede6b750"
Sep 18 16:01:22.919156 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:22.919119    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:27.151581 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:27.151551    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:28.777383 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:28.777346    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:28.789563 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:28.789530    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4"]
Sep 18 16:01:28.801324 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:28.801300    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-655d9c-29829120-wdpf4"]
Sep 18 16:01:28.827152 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:28.827120    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"]
Sep 18 16:01:28.827390 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:28.827362    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb" podUID="3e51525f-2ffb-46ae-90fa-ba7bee1a7832" containerName="payload-processing" containerID="cri-o://b11b284e9525d4c4f683320feddae6cd98a22a5aad2953c3ff981161f8476cd3" gracePeriod=30
Sep 18 16:01:28.836948 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:28.836866    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"]
Sep 18 16:01:28.837112 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:28.837087    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl" podUID="dfab3c15-ce3e-40b1-a9af-ca81282a5a2e" containerName="payload-pre-processing" containerID="cri-o://914e0529d5b145f2336ba4a109617ed72c90c3de99e76903324467792f4f8208" gracePeriod=30
Sep 18 16:01:28.913555 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:28.913526    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:29.047150 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.047113    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:29.119609 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.119576    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9"]
Sep 18 16:01:29.120544 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.120517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:29.120883 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.120848    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"
Sep 18 16:01:29.121784 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.121766    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-655d9c-fr8c9"]
Sep 18 16:01:29.124448 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.124432    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"
Sep 18 16:01:29.203965 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.203936    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86v56\" (UniqueName: \"kubernetes.io/projected/dfab3c15-ce3e-40b1-a9af-ca81282a5a2e-kube-api-access-86v56\") pod \"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e\" (UID: \"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e\") "
Sep 18 16:01:29.204136 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.203972    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4fpq\" (UniqueName: \"kubernetes.io/projected/3e51525f-2ffb-46ae-90fa-ba7bee1a7832-kube-api-access-p4fpq\") pod \"3e51525f-2ffb-46ae-90fa-ba7bee1a7832\" (UID: \"3e51525f-2ffb-46ae-90fa-ba7bee1a7832\") "
Sep 18 16:01:29.204136 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.204006    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e51525f-2ffb-46ae-90fa-ba7bee1a7832-plugins-config-volume\") pod \"3e51525f-2ffb-46ae-90fa-ba7bee1a7832\" (UID: \"3e51525f-2ffb-46ae-90fa-ba7bee1a7832\") "
Sep 18 16:01:29.204136 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.204031    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfab3c15-ce3e-40b1-a9af-ca81282a5a2e-plugins-config-volume\") pod \"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e\" (UID: \"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e\") "
Sep 18 16:01:29.204431 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.204402    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e51525f-2ffb-46ae-90fa-ba7bee1a7832-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e51525f-2ffb-46ae-90fa-ba7bee1a7832" (UID: "3e51525f-2ffb-46ae-90fa-ba7bee1a7832"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:01:29.204503 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.204440    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfab3c15-ce3e-40b1-a9af-ca81282a5a2e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dfab3c15-ce3e-40b1-a9af-ca81282a5a2e" (UID: "dfab3c15-ce3e-40b1-a9af-ca81282a5a2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:01:29.206199 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.206172    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfab3c15-ce3e-40b1-a9af-ca81282a5a2e-kube-api-access-86v56" (OuterVolumeSpecName: "kube-api-access-86v56") pod "dfab3c15-ce3e-40b1-a9af-ca81282a5a2e" (UID: "dfab3c15-ce3e-40b1-a9af-ca81282a5a2e"). InnerVolumeSpecName "kube-api-access-86v56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:01:29.206316 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.206292    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e51525f-2ffb-46ae-90fa-ba7bee1a7832-kube-api-access-p4fpq" (OuterVolumeSpecName: "kube-api-access-p4fpq") pod "3e51525f-2ffb-46ae-90fa-ba7bee1a7832" (UID: "3e51525f-2ffb-46ae-90fa-ba7bee1a7832"). InnerVolumeSpecName "kube-api-access-p4fpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:01:29.305179 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.305098    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86v56\" (UniqueName: \"kubernetes.io/projected/dfab3c15-ce3e-40b1-a9af-ca81282a5a2e-kube-api-access-86v56\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:01:29.305179 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.305126    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4fpq\" (UniqueName: \"kubernetes.io/projected/3e51525f-2ffb-46ae-90fa-ba7bee1a7832-kube-api-access-p4fpq\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:01:29.305179 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.305138    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e51525f-2ffb-46ae-90fa-ba7bee1a7832-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:01:29.305179 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.305150    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfab3c15-ce3e-40b1-a9af-ca81282a5a2e-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:01:29.493515 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.493460    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:29.592313 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.592223    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:29.673385 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.673347    2559 generic.go:356] "Generic (PLEG): container finished" podID="3e51525f-2ffb-46ae-90fa-ba7bee1a7832" containerID="b11b284e9525d4c4f683320feddae6cd98a22a5aad2953c3ff981161f8476cd3" exitCode=0
Sep 18 16:01:29.673535 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.673402    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"
Sep 18 16:01:29.673535 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.673434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb" event={"ID":"3e51525f-2ffb-46ae-90fa-ba7bee1a7832","Type":"ContainerDied","Data":"b11b284e9525d4c4f683320feddae6cd98a22a5aad2953c3ff981161f8476cd3"}
Sep 18 16:01:29.673535 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.673525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb" event={"ID":"3e51525f-2ffb-46ae-90fa-ba7bee1a7832","Type":"ContainerDied","Data":"f833388c357189fa36c4cc0cf9bd81e6939f8f2ed76e8c4d5cd553c8e7607cf0"}
Sep 18 16:01:29.673667 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.673564    2559 scope.go:117] "RemoveContainer" containerID="b11b284e9525d4c4f683320feddae6cd98a22a5aad2953c3ff981161f8476cd3"
Sep 18 16:01:29.674996 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.674973    2559 generic.go:356] "Generic (PLEG): container finished" podID="dfab3c15-ce3e-40b1-a9af-ca81282a5a2e" containerID="914e0529d5b145f2336ba4a109617ed72c90c3de99e76903324467792f4f8208" exitCode=0
Sep 18 16:01:29.675112 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.675022    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"
Sep 18 16:01:29.675112 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.675034    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl" event={"ID":"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e","Type":"ContainerDied","Data":"914e0529d5b145f2336ba4a109617ed72c90c3de99e76903324467792f4f8208"}
Sep 18 16:01:29.675112 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.675074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl" event={"ID":"dfab3c15-ce3e-40b1-a9af-ca81282a5a2e","Type":"ContainerDied","Data":"75abf22f78750c8eed8a7d3be3cfd5f2976c1acfb5fa895fd87923ef37ac40af"}
Sep 18 16:01:29.682810 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.682791    2559 scope.go:117] "RemoveContainer" containerID="b11b284e9525d4c4f683320feddae6cd98a22a5aad2953c3ff981161f8476cd3"
Sep 18 16:01:29.683117 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:01:29.683089    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b11b284e9525d4c4f683320feddae6cd98a22a5aad2953c3ff981161f8476cd3\": container with ID starting with b11b284e9525d4c4f683320feddae6cd98a22a5aad2953c3ff981161f8476cd3 not found: ID does not exist" containerID="b11b284e9525d4c4f683320feddae6cd98a22a5aad2953c3ff981161f8476cd3"
Sep 18 16:01:29.683329 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.683126    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b11b284e9525d4c4f683320feddae6cd98a22a5aad2953c3ff981161f8476cd3"} err="failed to get container status \"b11b284e9525d4c4f683320feddae6cd98a22a5aad2953c3ff981161f8476cd3\": rpc error: code = NotFound desc = could not find container \"b11b284e9525d4c4f683320feddae6cd98a22a5aad2953c3ff981161f8476cd3\": container with ID starting with b11b284e9525d4c4f683320feddae6cd98a22a5aad2953c3ff981161f8476cd3 not found: ID does not exist"
Sep 18 16:01:29.683329 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.683149    2559 scope.go:117] "RemoveContainer" containerID="914e0529d5b145f2336ba4a109617ed72c90c3de99e76903324467792f4f8208"
Sep 18 16:01:29.691024 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.691005    2559 scope.go:117] "RemoveContainer" containerID="914e0529d5b145f2336ba4a109617ed72c90c3de99e76903324467792f4f8208"
Sep 18 16:01:29.691270 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:01:29.691241    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914e0529d5b145f2336ba4a109617ed72c90c3de99e76903324467792f4f8208\": container with ID starting with 914e0529d5b145f2336ba4a109617ed72c90c3de99e76903324467792f4f8208 not found: ID does not exist" containerID="914e0529d5b145f2336ba4a109617ed72c90c3de99e76903324467792f4f8208"
Sep 18 16:01:29.691337 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.691276    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914e0529d5b145f2336ba4a109617ed72c90c3de99e76903324467792f4f8208"} err="failed to get container status \"914e0529d5b145f2336ba4a109617ed72c90c3de99e76903324467792f4f8208\": rpc error: code = NotFound desc = could not find container \"914e0529d5b145f2336ba4a109617ed72c90c3de99e76903324467792f4f8208\": container with ID starting with 914e0529d5b145f2336ba4a109617ed72c90c3de99e76903324467792f4f8208 not found: ID does not exist"
Sep 18 16:01:29.707132 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.707107    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"]
Sep 18 16:01:29.718603 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.718582    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-655d9c-6c4c9cf9f4-982nl"]
Sep 18 16:01:29.743495 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.743472    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aa23d8a-6133-409e-a8e7-41b438d06196" path="/var/lib/kubelet/pods/5aa23d8a-6133-409e-a8e7-41b438d06196/volumes"
Sep 18 16:01:29.743812 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.743800    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfab3c15-ce3e-40b1-a9af-ca81282a5a2e" path="/var/lib/kubelet/pods/dfab3c15-ce3e-40b1-a9af-ca81282a5a2e/volumes"
Sep 18 16:01:29.744063 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.744052    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec5a000-051e-47b7-ad97-d589cd533fe5" path="/var/lib/kubelet/pods/fec5a000-051e-47b7-ad97-d589cd533fe5/volumes"
Sep 18 16:01:29.756771 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.756749    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"]
Sep 18 16:01:29.766804 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:29.766780    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-655d9c-55595775f7-d7hfb"]
Sep 18 16:01:30.499850 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:30.499803    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:31.201908 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:31.201850    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:31.743707 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:31.743672    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e51525f-2ffb-46ae-90fa-ba7bee1a7832" path="/var/lib/kubelet/pods/3e51525f-2ffb-46ae-90fa-ba7bee1a7832/volumes"
Sep 18 16:01:32.281006 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:32.280975    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:01:35.625816 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:35.625775    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:35.731460 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:35.731427    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:35.922921 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:35.922815    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:01:40.719388 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:40.719358    2559 scope.go:117] "RemoveContainer" containerID="bf9d170c43d2fa31f0de8d2ffa95a26b3e5c802cf2ea4fae9f4e37697b47b502"
Sep 18 16:01:48.066333 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:01:48.066291    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:02:02.772065 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:02.772023    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:02:08.813349 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:08.813312    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:09.817170 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:09.817132    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:10.280417 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:10.280368    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:10.655635 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:10.655547    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:10.958932 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:10.958888    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:13.179265 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:13.179215    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:13.476396 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:13.476360    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:02:13.943240 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:13.943206    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:14.612376 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:14.612344    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:14.948355 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:14.948312    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:15.837218 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:15.837185    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:15.919440 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:15.919405    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:16.438663 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.438623    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:16.535579 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.535544    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:16.619107 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.619068    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"]
Sep 18 16:02:16.619466 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.619453    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfab3c15-ce3e-40b1-a9af-ca81282a5a2e" containerName="payload-pre-processing"
Sep 18 16:02:16.619511 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.619467    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfab3c15-ce3e-40b1-a9af-ca81282a5a2e" containerName="payload-pre-processing"
Sep 18 16:02:16.619511 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.619482    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa23d8a-6133-409e-a8e7-41b438d06196" containerName="revoke-keys"
Sep 18 16:02:16.619511 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.619487    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa23d8a-6133-409e-a8e7-41b438d06196" containerName="revoke-keys"
Sep 18 16:02:16.619511 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.619500    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e51525f-2ffb-46ae-90fa-ba7bee1a7832" containerName="payload-processing"
Sep 18 16:02:16.619511 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.619506    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e51525f-2ffb-46ae-90fa-ba7bee1a7832" containerName="payload-processing"
Sep 18 16:02:16.619667 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.619563    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aa23d8a-6133-409e-a8e7-41b438d06196" containerName="revoke-keys"
Sep 18 16:02:16.619667 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.619573    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfab3c15-ce3e-40b1-a9af-ca81282a5a2e" containerName="payload-pre-processing"
Sep 18 16:02:16.619667 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.619581    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e51525f-2ffb-46ae-90fa-ba7bee1a7832" containerName="payload-processing"
Sep 18 16:02:16.622812 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.622793    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"
Sep 18 16:02:16.627745 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.627721    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-90ebe5\""
Sep 18 16:02:16.627875 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.627722    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-90ebe5-dockercfg-snmx4\""
Sep 18 16:02:16.639871 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.639819    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ztmv\" (UniqueName: \"kubernetes.io/projected/11942848-e367-48a3-b6be-13b1bc6a1db5-kube-api-access-9ztmv\") pod \"payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86\" (UID: \"11942848-e367-48a3-b6be-13b1bc6a1db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"
Sep 18 16:02:16.640060 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.639992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11942848-e367-48a3-b6be-13b1bc6a1db5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86\" (UID: \"11942848-e367-48a3-b6be-13b1bc6a1db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"
Sep 18 16:02:16.640505 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.640487    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"]
Sep 18 16:02:16.664873 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.664822    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"]
Sep 18 16:02:16.668329 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.668303    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"
Sep 18 16:02:16.682466 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.682433    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"]
Sep 18 16:02:16.741137 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.741047    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mvsg\" (UniqueName: \"kubernetes.io/projected/65aa993d-9443-479c-87cd-2753bdb34546-kube-api-access-7mvsg\") pod \"payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8\" (UID: \"65aa993d-9443-479c-87cd-2753bdb34546\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"
Sep 18 16:02:16.741137 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.741094    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11942848-e367-48a3-b6be-13b1bc6a1db5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86\" (UID: \"11942848-e367-48a3-b6be-13b1bc6a1db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"
Sep 18 16:02:16.741331 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.741173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ztmv\" (UniqueName: \"kubernetes.io/projected/11942848-e367-48a3-b6be-13b1bc6a1db5-kube-api-access-9ztmv\") pod \"payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86\" (UID: \"11942848-e367-48a3-b6be-13b1bc6a1db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"
Sep 18 16:02:16.741331 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.741205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65aa993d-9443-479c-87cd-2753bdb34546-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8\" (UID: \"65aa993d-9443-479c-87cd-2753bdb34546\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"
Sep 18 16:02:16.741719 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.741702    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11942848-e367-48a3-b6be-13b1bc6a1db5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86\" (UID: \"11942848-e367-48a3-b6be-13b1bc6a1db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"
Sep 18 16:02:16.753235 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.753210    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ztmv\" (UniqueName: \"kubernetes.io/projected/11942848-e367-48a3-b6be-13b1bc6a1db5-kube-api-access-9ztmv\") pod \"payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86\" (UID: \"11942848-e367-48a3-b6be-13b1bc6a1db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"
Sep 18 16:02:16.842055 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.841995    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mvsg\" (UniqueName: \"kubernetes.io/projected/65aa993d-9443-479c-87cd-2753bdb34546-kube-api-access-7mvsg\") pod \"payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8\" (UID: \"65aa993d-9443-479c-87cd-2753bdb34546\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"
Sep 18 16:02:16.842452 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.842115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65aa993d-9443-479c-87cd-2753bdb34546-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8\" (UID: \"65aa993d-9443-479c-87cd-2753bdb34546\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"
Sep 18 16:02:16.842830 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.842809    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65aa993d-9443-479c-87cd-2753bdb34546-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8\" (UID: \"65aa993d-9443-479c-87cd-2753bdb34546\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"
Sep 18 16:02:16.856666 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.856630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mvsg\" (UniqueName: \"kubernetes.io/projected/65aa993d-9443-479c-87cd-2753bdb34546-kube-api-access-7mvsg\") pod \"payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8\" (UID: \"65aa993d-9443-479c-87cd-2753bdb34546\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"
Sep 18 16:02:16.934340 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.934297    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"
Sep 18 16:02:16.979972 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:16.979913    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"
Sep 18 16:02:17.209610 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:17.209582    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"]
Sep 18 16:02:17.211928 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:02:17.211903    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11942848_e367_48a3_b6be_13b1bc6a1db5.slice/crio-fe530c0d241e82a10451c42e9e518acb39e84ed68c20eb7861d5b112ddc98035 WatchSource:0}: Error finding container fe530c0d241e82a10451c42e9e518acb39e84ed68c20eb7861d5b112ddc98035: Status 404 returned error can't find the container with id fe530c0d241e82a10451c42e9e518acb39e84ed68c20eb7861d5b112ddc98035
Sep 18 16:02:17.264914 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:17.264888    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"]
Sep 18 16:02:17.265909 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:02:17.265885    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65aa993d_9443_479c_87cd_2753bdb34546.slice/crio-5bf604fbfebd560ad67ad77e7c3a6514476dddfaa34516883ba931a356d4a8c4 WatchSource:0}: Error finding container 5bf604fbfebd560ad67ad77e7c3a6514476dddfaa34516883ba931a356d4a8c4: Status 404 returned error can't find the container with id 5bf604fbfebd560ad67ad77e7c3a6514476dddfaa34516883ba931a356d4a8c4
Sep 18 16:02:17.924079 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:17.924041    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86" event={"ID":"11942848-e367-48a3-b6be-13b1bc6a1db5","Type":"ContainerStarted","Data":"3684295bed0569bb5b2776cad2285699dc3ab476b9ff092d7d10ff3c0a8c7dbd"}
Sep 18 16:02:17.924620 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:17.924207    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"
Sep 18 16:02:17.924620 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:17.924249    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86" event={"ID":"11942848-e367-48a3-b6be-13b1bc6a1db5","Type":"ContainerStarted","Data":"fe530c0d241e82a10451c42e9e518acb39e84ed68c20eb7861d5b112ddc98035"}
Sep 18 16:02:17.925584 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:17.925563    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8" event={"ID":"65aa993d-9443-479c-87cd-2753bdb34546","Type":"ContainerStarted","Data":"cce21599ec1c272ae3358526fd6f8668b7b7d63671812ac8994cbff26b325e88"}
Sep 18 16:02:17.925651 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:17.925599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8" event={"ID":"65aa993d-9443-479c-87cd-2753bdb34546","Type":"ContainerStarted","Data":"5bf604fbfebd560ad67ad77e7c3a6514476dddfaa34516883ba931a356d4a8c4"}
Sep 18 16:02:17.925651 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:17.925617    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"
Sep 18 16:02:17.944373 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:17.944324    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86" podStartSLOduration=1.944312355 podStartE2EDuration="1.944312355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:02:17.942227091 +0000 UTC m=+1660.503459236" watchObservedRunningTime="2026-09-18 16:02:17.944312355 +0000 UTC m=+1660.505544493"
Sep 18 16:02:17.963222 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:17.963169    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8" podStartSLOduration=1.9631519370000001 podStartE2EDuration="1.963151937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:02:17.960046019 +0000 UTC m=+1660.521278187" watchObservedRunningTime="2026-09-18 16:02:17.963151937 +0000 UTC m=+1660.524384079"
Sep 18 16:02:27.270503 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:27.270465    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:02:28.933636 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:28.933590    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"
Sep 18 16:02:28.934153 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:28.934008    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"
Sep 18 16:02:31.876896 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:31.876837    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:38.101689 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:38.101648    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:39.595115 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:39.595078    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:40.495538 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:40.495503    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:41.150895 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:41.150849    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:42.014593 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:42.014526    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:44.109849 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:44.109815    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:44.910195 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:44.910154    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:45.299122 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:45.299084    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:45.871307 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:45.871281    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:47.475824 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:47.475783    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:48.187142 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:48.187108    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:48.333512 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:48.333469    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:48.525018 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:48.524922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:49.684921 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:49.684885    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:50.433219 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:50.433185    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:51.199553 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.199475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:51.272130 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.272094    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:51.401788 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.401752    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:51.696838 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.696803    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:51.793221 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.793187    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:02:51.836416 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.836387    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"]
Sep 18 16:02:51.840074 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.840059    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"
Sep 18 16:02:51.843048 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.843028    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e41875\""
Sep 18 16:02:51.843181 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.843115    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e41875-dockercfg-7h5gb\""
Sep 18 16:02:51.867877 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.857061    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"]
Sep 18 16:02:51.876469 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.876260    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"]
Sep 18 16:02:51.879719 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.879701    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"
Sep 18 16:02:51.890722 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.890697    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"]
Sep 18 16:02:51.955123 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.955050    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05a44e79-492b-4246-9580-e89ce8be7b7b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s\" (UID: \"05a44e79-492b-4246-9580-e89ce8be7b7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"
Sep 18 16:02:51.955123 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.955114    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82740fe-d8fb-4276-943a-660e36ad05af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw\" (UID: \"e82740fe-d8fb-4276-943a-660e36ad05af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"
Sep 18 16:02:51.955307 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.955139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwtw\" (UniqueName: \"kubernetes.io/projected/e82740fe-d8fb-4276-943a-660e36ad05af-kube-api-access-slwtw\") pod \"payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw\" (UID: \"e82740fe-d8fb-4276-943a-660e36ad05af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"
Sep 18 16:02:51.955307 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:51.955191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj7b9\" (UniqueName: \"kubernetes.io/projected/05a44e79-492b-4246-9580-e89ce8be7b7b-kube-api-access-gj7b9\") pod \"payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s\" (UID: \"05a44e79-492b-4246-9580-e89ce8be7b7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"
Sep 18 16:02:52.056476 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:52.056440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82740fe-d8fb-4276-943a-660e36ad05af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw\" (UID: \"e82740fe-d8fb-4276-943a-660e36ad05af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"
Sep 18 16:02:52.056476 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:52.056476    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwtw\" (UniqueName: \"kubernetes.io/projected/e82740fe-d8fb-4276-943a-660e36ad05af-kube-api-access-slwtw\") pod \"payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw\" (UID: \"e82740fe-d8fb-4276-943a-660e36ad05af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"
Sep 18 16:02:52.056680 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:52.056606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj7b9\" (UniqueName: \"kubernetes.io/projected/05a44e79-492b-4246-9580-e89ce8be7b7b-kube-api-access-gj7b9\") pod \"payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s\" (UID: \"05a44e79-492b-4246-9580-e89ce8be7b7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"
Sep 18 16:02:52.056721 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:52.056690    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05a44e79-492b-4246-9580-e89ce8be7b7b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s\" (UID: \"05a44e79-492b-4246-9580-e89ce8be7b7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"
Sep 18 16:02:52.057400 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:52.057376    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82740fe-d8fb-4276-943a-660e36ad05af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw\" (UID: \"e82740fe-d8fb-4276-943a-660e36ad05af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"
Sep 18 16:02:52.057469 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:52.057455    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05a44e79-492b-4246-9580-e89ce8be7b7b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s\" (UID: \"05a44e79-492b-4246-9580-e89ce8be7b7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"
Sep 18 16:02:52.065749 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:52.065725    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwtw\" (UniqueName: \"kubernetes.io/projected/e82740fe-d8fb-4276-943a-660e36ad05af-kube-api-access-slwtw\") pod \"payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw\" (UID: \"e82740fe-d8fb-4276-943a-660e36ad05af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"
Sep 18 16:02:52.065909 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:52.065787    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj7b9\" (UniqueName: \"kubernetes.io/projected/05a44e79-492b-4246-9580-e89ce8be7b7b-kube-api-access-gj7b9\") pod \"payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s\" (UID: \"05a44e79-492b-4246-9580-e89ce8be7b7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"
Sep 18 16:02:52.151434 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:52.151408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"
Sep 18 16:02:52.191236 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:52.191204    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"
Sep 18 16:02:52.425337 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:52.425306    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"]
Sep 18 16:02:52.426844 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:02:52.426819    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a44e79_492b_4246_9580_e89ce8be7b7b.slice/crio-3a5c731ce416cb50b70b5f1e401fc66921c9f62f8c2c758c901dc2b12f9281b9 WatchSource:0}: Error finding container 3a5c731ce416cb50b70b5f1e401fc66921c9f62f8c2c758c901dc2b12f9281b9: Status 404 returned error can't find the container with id 3a5c731ce416cb50b70b5f1e401fc66921c9f62f8c2c758c901dc2b12f9281b9
Sep 18 16:02:52.474940 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:52.474916    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"]
Sep 18 16:02:52.476565 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:02:52.476541    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82740fe_d8fb_4276_943a_660e36ad05af.slice/crio-60b217f50a8f80d134072ed11a75f68afbf85fe1d30ae8eddbf61ca1d9909c29 WatchSource:0}: Error finding container 60b217f50a8f80d134072ed11a75f68afbf85fe1d30ae8eddbf61ca1d9909c29: Status 404 returned error can't find the container with id 60b217f50a8f80d134072ed11a75f68afbf85fe1d30ae8eddbf61ca1d9909c29
Sep 18 16:02:53.110242 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:53.110198    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s" event={"ID":"05a44e79-492b-4246-9580-e89ce8be7b7b","Type":"ContainerStarted","Data":"ae75e18badcc50499d6de42dd67db58afecdc0138834f73ee1e4dd89414d93af"}
Sep 18 16:02:53.110431 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:53.110382    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"
Sep 18 16:02:53.110502 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:53.110431    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s" event={"ID":"05a44e79-492b-4246-9580-e89ce8be7b7b","Type":"ContainerStarted","Data":"3a5c731ce416cb50b70b5f1e401fc66921c9f62f8c2c758c901dc2b12f9281b9"}
Sep 18 16:02:53.111819 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:53.111794    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw" event={"ID":"e82740fe-d8fb-4276-943a-660e36ad05af","Type":"ContainerStarted","Data":"03f4f5c034602985e3f0a56071041f864084ebcf2fc631d36b7029b2467e2aab"}
Sep 18 16:02:53.111940 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:53.111831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw" event={"ID":"e82740fe-d8fb-4276-943a-660e36ad05af","Type":"ContainerStarted","Data":"60b217f50a8f80d134072ed11a75f68afbf85fe1d30ae8eddbf61ca1d9909c29"}
Sep 18 16:02:53.111940 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:53.111873    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"
Sep 18 16:02:53.128734 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:53.128677    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s" podStartSLOduration=2.128664257 podStartE2EDuration="2.128664257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:02:53.127401394 +0000 UTC m=+1695.688633527" watchObservedRunningTime="2026-09-18 16:02:53.128664257 +0000 UTC m=+1695.689896394"
Sep 18 16:02:53.146759 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:53.146720    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw" podStartSLOduration=2.146706121 podStartE2EDuration="2.146706121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:02:53.145367688 +0000 UTC m=+1695.706599827" watchObservedRunningTime="2026-09-18 16:02:53.146706121 +0000 UTC m=+1695.707938260"
Sep 18 16:02:55.084261 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:02:55.084229    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:03:04.119258 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:04.119227    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"
Sep 18 16:03:04.119769 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:04.119583    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"
Sep 18 16:03:08.420720 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:08.420681    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:03:13.117879 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:13.117834    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:03:13.415415 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:13.415329    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:03:14.190735 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:14.190699    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:03:15.190505 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:15.190470    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:03:15.830060 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:15.830019    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:03:16.648803 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:16.648765    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:03:18.154723 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:18.154699    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-mx9mf_619761f1-b0a4-4b32-978d-5c3702133c7d/payload-processing/0.log"
Sep 18 16:03:18.290657 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:18.290625    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-mx9mf_619761f1-b0a4-4b32-978d-5c3702133c7d/payload-processing/0.log"
Sep 18 16:03:18.454351 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:18.454321    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s_05a44e79-492b-4246-9580-e89ce8be7b7b/payload-processing/0.log"
Sep 18 16:03:29.786775 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:29.786738    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:03:31.359603 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:31.359562    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:03:35.279676 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:35.279643    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:03:40.777693 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:40.777661    2559 scope.go:117] "RemoveContainer" containerID="4ddd8e4e1e7682de330130168868928459a9f999097fc91a68c2efc83df724d5"
Sep 18 16:03:40.788086 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:40.788065    2559 scope.go:117] "RemoveContainer" containerID="efc273030db7c1e370912a4b0f994d3b19be78762d0d4f4f6509b477e977a961"
Sep 18 16:03:52.966367 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:52.966331    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:03:56.583706 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:03:56.583668    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:00.506751 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:00.506708    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:00.585484 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:00.585456    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:03.882745 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:03.882712    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:04:09.899411 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:09.899377    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8_65aa993d-9443-479c-87cd-2753bdb34546/payload-processing/0.log"
Sep 18 16:04:10.014936 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:10.014905    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-mx9mf_619761f1-b0a4-4b32-978d-5c3702133c7d/payload-processing/0.log"
Sep 18 16:04:10.134841 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:10.134816    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-mx9mf_619761f1-b0a4-4b32-978d-5c3702133c7d/payload-processing/0.log"
Sep 18 16:04:10.245984 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:10.245963    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s_05a44e79-492b-4246-9580-e89ce8be7b7b/payload-processing/0.log"
Sep 18 16:04:10.371093 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:10.371068    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s_05a44e79-492b-4246-9580-e89ce8be7b7b/payload-processing/0.log"
Sep 18 16:04:10.652651 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:10.652562    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:10.957493 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:10.957458    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:11.338169 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:11.338075    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:13.670192 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:13.670161    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:14.281174 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:14.281135    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:14.457398 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:14.457360    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:14.556560 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:14.556470    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:15.430737 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:15.430708    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:16.177161 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:16.177125    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:17.121027 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:17.120992    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:18.148212 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:18.148175    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:18.249446 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:18.249412    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:18.763220 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:18.763184    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:18.862620 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:18.862578    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:18.905834 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:18.905804    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"]
Sep 18 16:04:18.908870 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:18.908838    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"
Sep 18 16:04:18.913313 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:18.913290    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-e2585e\""
Sep 18 16:04:18.913420 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:18.913379    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-e2585e-dockercfg-x56vx\""
Sep 18 16:04:18.929971 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:18.929945    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"]
Sep 18 16:04:18.971869 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:18.971831    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"]
Sep 18 16:04:18.974478 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:18.974456    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"
Sep 18 16:04:18.993210 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:18.993189    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"]
Sep 18 16:04:19.039525 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.039438    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bw2r\" (UniqueName: \"kubernetes.io/projected/336d98c2-919b-4738-aa95-ad16532827b9-kube-api-access-6bw2r\") pod \"payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb\" (UID: \"336d98c2-919b-4738-aa95-ad16532827b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"
Sep 18 16:04:19.039525 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.039478    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336d98c2-919b-4738-aa95-ad16532827b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb\" (UID: \"336d98c2-919b-4738-aa95-ad16532827b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"
Sep 18 16:04:19.039723 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.039553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpq4m\" (UniqueName: \"kubernetes.io/projected/3407314a-8c7c-4e03-9851-f70100e199b9-kube-api-access-gpq4m\") pod \"payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx\" (UID: \"3407314a-8c7c-4e03-9851-f70100e199b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"
Sep 18 16:04:19.039723 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.039592    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3407314a-8c7c-4e03-9851-f70100e199b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx\" (UID: \"3407314a-8c7c-4e03-9851-f70100e199b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"
Sep 18 16:04:19.140259 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.140225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3407314a-8c7c-4e03-9851-f70100e199b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx\" (UID: \"3407314a-8c7c-4e03-9851-f70100e199b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"
Sep 18 16:04:19.140438 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.140305    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bw2r\" (UniqueName: \"kubernetes.io/projected/336d98c2-919b-4738-aa95-ad16532827b9-kube-api-access-6bw2r\") pod \"payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb\" (UID: \"336d98c2-919b-4738-aa95-ad16532827b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"
Sep 18 16:04:19.140438 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.140333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336d98c2-919b-4738-aa95-ad16532827b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb\" (UID: \"336d98c2-919b-4738-aa95-ad16532827b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"
Sep 18 16:04:19.140438 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.140385    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpq4m\" (UniqueName: \"kubernetes.io/projected/3407314a-8c7c-4e03-9851-f70100e199b9-kube-api-access-gpq4m\") pod \"payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx\" (UID: \"3407314a-8c7c-4e03-9851-f70100e199b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"
Sep 18 16:04:19.140983 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.140964    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3407314a-8c7c-4e03-9851-f70100e199b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx\" (UID: \"3407314a-8c7c-4e03-9851-f70100e199b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"
Sep 18 16:04:19.141052 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.141036    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336d98c2-919b-4738-aa95-ad16532827b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb\" (UID: \"336d98c2-919b-4738-aa95-ad16532827b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"
Sep 18 16:04:19.149101 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.149079    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpq4m\" (UniqueName: \"kubernetes.io/projected/3407314a-8c7c-4e03-9851-f70100e199b9-kube-api-access-gpq4m\") pod \"payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx\" (UID: \"3407314a-8c7c-4e03-9851-f70100e199b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"
Sep 18 16:04:19.149101 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.149101    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bw2r\" (UniqueName: \"kubernetes.io/projected/336d98c2-919b-4738-aa95-ad16532827b9-kube-api-access-6bw2r\") pod \"payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb\" (UID: \"336d98c2-919b-4738-aa95-ad16532827b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"
Sep 18 16:04:19.218551 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.218528    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"
Sep 18 16:04:19.283723 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.283692    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"
Sep 18 16:04:19.488191 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.488167    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"]
Sep 18 16:04:19.490046 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:04:19.490020    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336d98c2_919b_4738_aa95_ad16532827b9.slice/crio-57fa1af9dde78a65193774be5be8f1fb04eea0b3a79299815635f4479445961d WatchSource:0}: Error finding container 57fa1af9dde78a65193774be5be8f1fb04eea0b3a79299815635f4479445961d: Status 404 returned error can't find the container with id 57fa1af9dde78a65193774be5be8f1fb04eea0b3a79299815635f4479445961d
Sep 18 16:04:19.554502 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.554472    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"]
Sep 18 16:04:19.555450 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:04:19.555427    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3407314a_8c7c_4e03_9851_f70100e199b9.slice/crio-8f1e0251f761ce1cb582dfa2897a8f43c790cd18c3d3a9b58dce8501911b5da4 WatchSource:0}: Error finding container 8f1e0251f761ce1cb582dfa2897a8f43c790cd18c3d3a9b58dce8501911b5da4: Status 404 returned error can't find the container with id 8f1e0251f761ce1cb582dfa2897a8f43c790cd18c3d3a9b58dce8501911b5da4
Sep 18 16:04:19.577677 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.577649    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb" event={"ID":"336d98c2-919b-4738-aa95-ad16532827b9","Type":"ContainerStarted","Data":"b4cea9cce9c609b8824799d3d58c5c67b25ea4ee8bc95d2ee387c99fbc172a07"}
Sep 18 16:04:19.577785 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.577706    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb" event={"ID":"336d98c2-919b-4738-aa95-ad16532827b9","Type":"ContainerStarted","Data":"57fa1af9dde78a65193774be5be8f1fb04eea0b3a79299815635f4479445961d"}
Sep 18 16:04:19.577878 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.577810    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"
Sep 18 16:04:19.578981 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.578959    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx" event={"ID":"3407314a-8c7c-4e03-9851-f70100e199b9","Type":"ContainerStarted","Data":"8f1e0251f761ce1cb582dfa2897a8f43c790cd18c3d3a9b58dce8501911b5da4"}
Sep 18 16:04:19.600740 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:19.600691    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb" podStartSLOduration=1.600674375 podStartE2EDuration="1.600674375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:04:19.598119956 +0000 UTC m=+1782.159352096" watchObservedRunningTime="2026-09-18 16:04:19.600674375 +0000 UTC m=+1782.161906516"
Sep 18 16:04:20.585456 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:20.585416    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx" event={"ID":"3407314a-8c7c-4e03-9851-f70100e199b9","Type":"ContainerStarted","Data":"b588ddb1152d3e31081d6904f68326881bb3bc75ad7ab0fe88b9d609d7ef9c98"}
Sep 18 16:04:20.586000 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:20.585662    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"
Sep 18 16:04:20.606986 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:20.606928    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx" podStartSLOduration=2.606911836 podStartE2EDuration="2.606911836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:04:20.605845015 +0000 UTC m=+1783.167077183" watchObservedRunningTime="2026-09-18 16:04:20.606911836 +0000 UTC m=+1783.168143977"
Sep 18 16:04:23.678875 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:23.678758    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:04:24.848321 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:24.848283    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:30.587362 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:30.587330    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"
Sep 18 16:04:31.593549 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:31.593522    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"
Sep 18 16:04:41.894668 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:41.894625    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:42.717272 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:42.717232    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:43.486709 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:43.486677    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:44.226276 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:44.226245    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:44.584722 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:44.584618    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:45.052792 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:45.052752    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:45.287437 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:45.287400    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk"]
Sep 18 16:04:45.291186 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:45.291165    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk"
Sep 18 16:04:45.295826 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:45.295802    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 16:04:45.295973 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:45.295907    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:04:45.305772 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:45.305703    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk"]
Sep 18 16:04:45.372958 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:45.372929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab73432-298e-45b5-9e30-59260d084f9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk\" (UID: \"bab73432-298e-45b5-9e30-59260d084f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk"
Sep 18 16:04:45.474012 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:45.473979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab73432-298e-45b5-9e30-59260d084f9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk\" (UID: \"bab73432-298e-45b5-9e30-59260d084f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk"
Sep 18 16:04:45.474571 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:45.474552    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab73432-298e-45b5-9e30-59260d084f9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk\" (UID: \"bab73432-298e-45b5-9e30-59260d084f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk"
Sep 18 16:04:45.600685 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:45.600605    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk"
Sep 18 16:04:45.876878 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:45.876753    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk"]
Sep 18 16:04:45.879588 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:04:45.879560    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab73432_298e_45b5_9e30_59260d084f9e.slice/crio-d0100043ffac2c090c956e71832bf7c1c39ba464c8c9af93ad054b2b7eb46b1e WatchSource:0}: Error finding container d0100043ffac2c090c956e71832bf7c1c39ba464c8c9af93ad054b2b7eb46b1e: Status 404 returned error can't find the container with id d0100043ffac2c090c956e71832bf7c1c39ba464c8c9af93ad054b2b7eb46b1e
Sep 18 16:04:45.972563 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:45.972529    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:46.714287 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:46.714256    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:46.725797 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:46.725769    2559 generic.go:356] "Generic (PLEG): container finished" podID="bab73432-298e-45b5-9e30-59260d084f9e" containerID="71994f07b0ce015ff68217f991851a2d0d6731ebbc2ea78eafa05be6dcb3fba7" exitCode=0
Sep 18 16:04:46.725943 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:46.725863    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk" event={"ID":"bab73432-298e-45b5-9e30-59260d084f9e","Type":"ContainerDied","Data":"71994f07b0ce015ff68217f991851a2d0d6731ebbc2ea78eafa05be6dcb3fba7"}
Sep 18 16:04:46.725943 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:46.725917    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk" event={"ID":"bab73432-298e-45b5-9e30-59260d084f9e","Type":"ContainerStarted","Data":"d0100043ffac2c090c956e71832bf7c1c39ba464c8c9af93ad054b2b7eb46b1e"}
Sep 18 16:04:47.419890 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:47.419845    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:47.868048 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:47.868025    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk"
Sep 18 16:04:47.894841 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:47.894817    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab73432-298e-45b5-9e30-59260d084f9e-maas-api-service-ca\") pod \"bab73432-298e-45b5-9e30-59260d084f9e\" (UID: \"bab73432-298e-45b5-9e30-59260d084f9e\") "
Sep 18 16:04:47.895156 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:47.895138    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab73432-298e-45b5-9e30-59260d084f9e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bab73432-298e-45b5-9e30-59260d084f9e" (UID: "bab73432-298e-45b5-9e30-59260d084f9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:04:47.996270 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:47.996191    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab73432-298e-45b5-9e30-59260d084f9e-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:04:48.171808 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:48.171774    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:48.737540 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:48.737506    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk"
Sep 18 16:04:48.737540 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:48.737524    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk" event={"ID":"bab73432-298e-45b5-9e30-59260d084f9e","Type":"ContainerDied","Data":"d0100043ffac2c090c956e71832bf7c1c39ba464c8c9af93ad054b2b7eb46b1e"}
Sep 18 16:04:48.737751 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:48.737561    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0100043ffac2c090c956e71832bf7c1c39ba464c8c9af93ad054b2b7eb46b1e"
Sep 18 16:04:49.007301 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:49.007222    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:49.774960 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:49.774929    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:55.804966 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:55.804912    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:56.541758 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.541726    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:56.576846 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.576811    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"]
Sep 18 16:04:56.577307 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.577272    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb" podUID="336d98c2-919b-4738-aa95-ad16532827b9" containerName="payload-processing" containerID="cri-o://b4cea9cce9c609b8824799d3d58c5c67b25ea4ee8bc95d2ee387c99fbc172a07" gracePeriod=30
Sep 18 16:04:56.588455 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.588426    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"]
Sep 18 16:04:56.588662 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.588637    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx" podUID="3407314a-8c7c-4e03-9851-f70100e199b9" containerName="payload-pre-processing" containerID="cri-o://b588ddb1152d3e31081d6904f68326881bb3bc75ad7ab0fe88b9d609d7ef9c98" gracePeriod=30
Sep 18 16:04:56.629389 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.629351    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:56.703821 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.703790    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:56.784381 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.784351    2559 generic.go:356] "Generic (PLEG): container finished" podID="336d98c2-919b-4738-aa95-ad16532827b9" containerID="b4cea9cce9c609b8824799d3d58c5c67b25ea4ee8bc95d2ee387c99fbc172a07" exitCode=0
Sep 18 16:04:56.784553 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.784419    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb" event={"ID":"336d98c2-919b-4738-aa95-ad16532827b9","Type":"ContainerDied","Data":"b4cea9cce9c609b8824799d3d58c5c67b25ea4ee8bc95d2ee387c99fbc172a07"}
Sep 18 16:04:56.786428 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.786401    2559 generic.go:356] "Generic (PLEG): container finished" podID="3407314a-8c7c-4e03-9851-f70100e199b9" containerID="b588ddb1152d3e31081d6904f68326881bb3bc75ad7ab0fe88b9d609d7ef9c98" exitCode=0
Sep 18 16:04:56.786629 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.786458    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx" event={"ID":"3407314a-8c7c-4e03-9851-f70100e199b9","Type":"ContainerDied","Data":"b588ddb1152d3e31081d6904f68326881bb3bc75ad7ab0fe88b9d609d7ef9c98"}
Sep 18 16:04:56.787694 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.787671    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:56.808743 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.808716    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk"]
Sep 18 16:04:56.813929 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.813906    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e2585e-mx6lk"]
Sep 18 16:04:56.841584 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.841562    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"
Sep 18 16:04:56.863282 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.863058    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"
Sep 18 16:04:56.864588 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.864549    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:56.973054 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.973025    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpq4m\" (UniqueName: \"kubernetes.io/projected/3407314a-8c7c-4e03-9851-f70100e199b9-kube-api-access-gpq4m\") pod \"3407314a-8c7c-4e03-9851-f70100e199b9\" (UID: \"3407314a-8c7c-4e03-9851-f70100e199b9\") "
Sep 18 16:04:56.973251 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.973130    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336d98c2-919b-4738-aa95-ad16532827b9-plugins-config-volume\") pod \"336d98c2-919b-4738-aa95-ad16532827b9\" (UID: \"336d98c2-919b-4738-aa95-ad16532827b9\") "
Sep 18 16:04:56.973251 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.973151    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bw2r\" (UniqueName: \"kubernetes.io/projected/336d98c2-919b-4738-aa95-ad16532827b9-kube-api-access-6bw2r\") pod \"336d98c2-919b-4738-aa95-ad16532827b9\" (UID: \"336d98c2-919b-4738-aa95-ad16532827b9\") "
Sep 18 16:04:56.973251 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.973191    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3407314a-8c7c-4e03-9851-f70100e199b9-plugins-config-volume\") pod \"3407314a-8c7c-4e03-9851-f70100e199b9\" (UID: \"3407314a-8c7c-4e03-9851-f70100e199b9\") "
Sep 18 16:04:56.973578 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.973547    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336d98c2-919b-4738-aa95-ad16532827b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "336d98c2-919b-4738-aa95-ad16532827b9" (UID: "336d98c2-919b-4738-aa95-ad16532827b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:04:56.973578 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.973558    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3407314a-8c7c-4e03-9851-f70100e199b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3407314a-8c7c-4e03-9851-f70100e199b9" (UID: "3407314a-8c7c-4e03-9851-f70100e199b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:04:56.975152 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.975129    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3407314a-8c7c-4e03-9851-f70100e199b9-kube-api-access-gpq4m" (OuterVolumeSpecName: "kube-api-access-gpq4m") pod "3407314a-8c7c-4e03-9851-f70100e199b9" (UID: "3407314a-8c7c-4e03-9851-f70100e199b9"). InnerVolumeSpecName "kube-api-access-gpq4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:04:56.975227 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:56.975195    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336d98c2-919b-4738-aa95-ad16532827b9-kube-api-access-6bw2r" (OuterVolumeSpecName: "kube-api-access-6bw2r") pod "336d98c2-919b-4738-aa95-ad16532827b9" (UID: "336d98c2-919b-4738-aa95-ad16532827b9"). InnerVolumeSpecName "kube-api-access-6bw2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:04:57.074448 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.074372    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336d98c2-919b-4738-aa95-ad16532827b9-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:04:57.074448 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.074396    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bw2r\" (UniqueName: \"kubernetes.io/projected/336d98c2-919b-4738-aa95-ad16532827b9-kube-api-access-6bw2r\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:04:57.074448 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.074406    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3407314a-8c7c-4e03-9851-f70100e199b9-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:04:57.074448 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.074415    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpq4m\" (UniqueName: \"kubernetes.io/projected/3407314a-8c7c-4e03-9851-f70100e199b9-kube-api-access-gpq4m\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:04:57.190087 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.190051    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:57.292301 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.292261    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:57.744991 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.744963    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bab73432-298e-45b5-9e30-59260d084f9e" path="/var/lib/kubelet/pods/bab73432-298e-45b5-9e30-59260d084f9e/volumes"
Sep 18 16:04:57.793081 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.793052    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"
Sep 18 16:04:57.793081 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.793058    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb" event={"ID":"336d98c2-919b-4738-aa95-ad16532827b9","Type":"ContainerDied","Data":"57fa1af9dde78a65193774be5be8f1fb04eea0b3a79299815635f4479445961d"}
Sep 18 16:04:57.793342 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.793124    2559 scope.go:117] "RemoveContainer" containerID="b4cea9cce9c609b8824799d3d58c5c67b25ea4ee8bc95d2ee387c99fbc172a07"
Sep 18 16:04:57.794662 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.794639    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"
Sep 18 16:04:57.794799 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.794704    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx" event={"ID":"3407314a-8c7c-4e03-9851-f70100e199b9","Type":"ContainerDied","Data":"8f1e0251f761ce1cb582dfa2897a8f43c790cd18c3d3a9b58dce8501911b5da4"}
Sep 18 16:04:57.801948 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.801932    2559 scope.go:117] "RemoveContainer" containerID="b588ddb1152d3e31081d6904f68326881bb3bc75ad7ab0fe88b9d609d7ef9c98"
Sep 18 16:04:57.815458 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.815435    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"]
Sep 18 16:04:57.817194 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.817173    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e2585e-6f47579dfd-8fqgx"]
Sep 18 16:04:57.828095 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.828063    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"]
Sep 18 16:04:57.830098 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:57.830076    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e2585e-65896fc759-g28hb"]
Sep 18 16:04:58.050634 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:58.050604    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:58.918793 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:58.918748    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:59.709449 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:59.709408    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:04:59.743798 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:59.743761    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="336d98c2-919b-4738-aa95-ad16532827b9" path="/var/lib/kubelet/pods/336d98c2-919b-4738-aa95-ad16532827b9/volumes"
Sep 18 16:04:59.744207 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:04:59.744186    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3407314a-8c7c-4e03-9851-f70100e199b9" path="/var/lib/kubelet/pods/3407314a-8c7c-4e03-9851-f70100e199b9/volumes"
Sep 18 16:05:00.617380 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:00.617332    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:01.014251 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:01.014218    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:05:03.130344 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:03.130304    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:03.230211 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:03.230176    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:03.502361 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:03.502327    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:07.190772 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.190735    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j"]
Sep 18 16:05:07.191192 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.191178    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336d98c2-919b-4738-aa95-ad16532827b9" containerName="payload-processing"
Sep 18 16:05:07.191236 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.191193    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="336d98c2-919b-4738-aa95-ad16532827b9" containerName="payload-processing"
Sep 18 16:05:07.191236 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.191201    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bab73432-298e-45b5-9e30-59260d084f9e" containerName="revoke-keys"
Sep 18 16:05:07.191236 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.191206    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bab73432-298e-45b5-9e30-59260d084f9e" containerName="revoke-keys"
Sep 18 16:05:07.191236 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.191214    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3407314a-8c7c-4e03-9851-f70100e199b9" containerName="payload-pre-processing"
Sep 18 16:05:07.191236 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.191220    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3407314a-8c7c-4e03-9851-f70100e199b9" containerName="payload-pre-processing"
Sep 18 16:05:07.191389 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.191286    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bab73432-298e-45b5-9e30-59260d084f9e" containerName="revoke-keys"
Sep 18 16:05:07.191389 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.191297    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3407314a-8c7c-4e03-9851-f70100e199b9" containerName="payload-pre-processing"
Sep 18 16:05:07.191389 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.191303    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="336d98c2-919b-4738-aa95-ad16532827b9" containerName="payload-processing"
Sep 18 16:05:07.194005 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.193990    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j"
Sep 18 16:05:07.198081 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.198058    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 16:05:07.198362 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.198344    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:05:07.205007 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.204987    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j"]
Sep 18 16:05:07.264349 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.264324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da6f2f3-bfca-4327-bee7-9cb7c069a943-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j\" (UID: \"2da6f2f3-bfca-4327-bee7-9cb7c069a943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j"
Sep 18 16:05:07.323009 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.322978    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:07.365274 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.365248    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da6f2f3-bfca-4327-bee7-9cb7c069a943-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j\" (UID: \"2da6f2f3-bfca-4327-bee7-9cb7c069a943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j"
Sep 18 16:05:07.365844 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.365824    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da6f2f3-bfca-4327-bee7-9cb7c069a943-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j\" (UID: \"2da6f2f3-bfca-4327-bee7-9cb7c069a943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j"
Sep 18 16:05:07.504491 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.504411    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j"
Sep 18 16:05:07.772294 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.772267    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j"]
Sep 18 16:05:07.773951 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:05:07.773924    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da6f2f3_bfca_4327_bee7_9cb7c069a943.slice/crio-222c295d69367a5168684f3b6c4af0ec1966e71079c0091557e02620cf587ffe WatchSource:0}: Error finding container 222c295d69367a5168684f3b6c4af0ec1966e71079c0091557e02620cf587ffe: Status 404 returned error can't find the container with id 222c295d69367a5168684f3b6c4af0ec1966e71079c0091557e02620cf587ffe
Sep 18 16:05:07.849174 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:07.849146    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j" event={"ID":"2da6f2f3-bfca-4327-bee7-9cb7c069a943","Type":"ContainerStarted","Data":"222c295d69367a5168684f3b6c4af0ec1966e71079c0091557e02620cf587ffe"}
Sep 18 16:05:08.125784 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:08.125706    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:08.730811 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:08.730773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:08.856470 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:08.856415    2559 generic.go:356] "Generic (PLEG): container finished" podID="2da6f2f3-bfca-4327-bee7-9cb7c069a943" containerID="3e9615e7d2c766c13f46eb6e7ac8b80da709c27fbbcafb8e2687b677ad020ead" exitCode=0
Sep 18 16:05:08.856787 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:08.856574    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j" event={"ID":"2da6f2f3-bfca-4327-bee7-9cb7c069a943","Type":"ContainerDied","Data":"3e9615e7d2c766c13f46eb6e7ac8b80da709c27fbbcafb8e2687b677ad020ead"}
Sep 18 16:05:09.686581 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:09.686540    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:10.002239 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:10.002208    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j"
Sep 18 16:05:10.091099 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:10.091069    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da6f2f3-bfca-4327-bee7-9cb7c069a943-maas-api-service-ca\") pod \"2da6f2f3-bfca-4327-bee7-9cb7c069a943\" (UID: \"2da6f2f3-bfca-4327-bee7-9cb7c069a943\") "
Sep 18 16:05:10.091382 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:10.091360    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da6f2f3-bfca-4327-bee7-9cb7c069a943-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2da6f2f3-bfca-4327-bee7-9cb7c069a943" (UID: "2da6f2f3-bfca-4327-bee7-9cb7c069a943"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:05:10.192244 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:10.192212    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da6f2f3-bfca-4327-bee7-9cb7c069a943-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:05:10.484695 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:10.484661    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:10.870036 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:10.869949    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j"
Sep 18 16:05:10.870036 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:10.869978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j" event={"ID":"2da6f2f3-bfca-4327-bee7-9cb7c069a943","Type":"ContainerDied","Data":"222c295d69367a5168684f3b6c4af0ec1966e71079c0091557e02620cf587ffe"}
Sep 18 16:05:10.870036 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:10.870025    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="222c295d69367a5168684f3b6c4af0ec1966e71079c0091557e02620cf587ffe"
Sep 18 16:05:11.102714 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:11.102678    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:20.674423 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:20.674387    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:05:22.921328 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:22.921290    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:22.955304 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:22.955265    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"]
Sep 18 16:05:22.955558 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:22.955530    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8" podUID="65aa993d-9443-479c-87cd-2753bdb34546" containerName="payload-processing" containerID="cri-o://cce21599ec1c272ae3358526fd6f8668b7b7d63671812ac8994cbff26b325e88" gracePeriod=30
Sep 18 16:05:22.966933 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:22.966416    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"]
Sep 18 16:05:22.967351 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:22.967283    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86" podUID="11942848-e367-48a3-b6be-13b1bc6a1db5" containerName="payload-pre-processing" containerID="cri-o://3684295bed0569bb5b2776cad2285699dc3ab476b9ff092d7d10ff3c0a8c7dbd" gracePeriod=30
Sep 18 16:05:23.215521 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.215491    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"
Sep 18 16:05:23.250479 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.250444    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"
Sep 18 16:05:23.302129 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.302085    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j"]
Sep 18 16:05:23.310589 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.310554    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-90ebe5-7k56j"]
Sep 18 16:05:23.313688 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.313665    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mvsg\" (UniqueName: \"kubernetes.io/projected/65aa993d-9443-479c-87cd-2753bdb34546-kube-api-access-7mvsg\") pod \"65aa993d-9443-479c-87cd-2753bdb34546\" (UID: \"65aa993d-9443-479c-87cd-2753bdb34546\") "
Sep 18 16:05:23.313846 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.313706    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65aa993d-9443-479c-87cd-2753bdb34546-plugins-config-volume\") pod \"65aa993d-9443-479c-87cd-2753bdb34546\" (UID: \"65aa993d-9443-479c-87cd-2753bdb34546\") "
Sep 18 16:05:23.313846 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.313742    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11942848-e367-48a3-b6be-13b1bc6a1db5-plugins-config-volume\") pod \"11942848-e367-48a3-b6be-13b1bc6a1db5\" (UID: \"11942848-e367-48a3-b6be-13b1bc6a1db5\") "
Sep 18 16:05:23.313991 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.313885    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ztmv\" (UniqueName: \"kubernetes.io/projected/11942848-e367-48a3-b6be-13b1bc6a1db5-kube-api-access-9ztmv\") pod \"11942848-e367-48a3-b6be-13b1bc6a1db5\" (UID: \"11942848-e367-48a3-b6be-13b1bc6a1db5\") "
Sep 18 16:05:23.314193 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.314166    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65aa993d-9443-479c-87cd-2753bdb34546-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65aa993d-9443-479c-87cd-2753bdb34546" (UID: "65aa993d-9443-479c-87cd-2753bdb34546"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:05:23.314262 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.314171    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11942848-e367-48a3-b6be-13b1bc6a1db5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11942848-e367-48a3-b6be-13b1bc6a1db5" (UID: "11942848-e367-48a3-b6be-13b1bc6a1db5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:05:23.316005 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.315981    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11942848-e367-48a3-b6be-13b1bc6a1db5-kube-api-access-9ztmv" (OuterVolumeSpecName: "kube-api-access-9ztmv") pod "11942848-e367-48a3-b6be-13b1bc6a1db5" (UID: "11942848-e367-48a3-b6be-13b1bc6a1db5"). InnerVolumeSpecName "kube-api-access-9ztmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:05:23.316068 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.316000    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65aa993d-9443-479c-87cd-2753bdb34546-kube-api-access-7mvsg" (OuterVolumeSpecName: "kube-api-access-7mvsg") pod "65aa993d-9443-479c-87cd-2753bdb34546" (UID: "65aa993d-9443-479c-87cd-2753bdb34546"). InnerVolumeSpecName "kube-api-access-7mvsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:05:23.414665 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.414627    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11942848-e367-48a3-b6be-13b1bc6a1db5-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:05:23.414665 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.414664    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ztmv\" (UniqueName: \"kubernetes.io/projected/11942848-e367-48a3-b6be-13b1bc6a1db5-kube-api-access-9ztmv\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:05:23.414665 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.414673    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mvsg\" (UniqueName: \"kubernetes.io/projected/65aa993d-9443-479c-87cd-2753bdb34546-kube-api-access-7mvsg\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:05:23.414952 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.414682    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65aa993d-9443-479c-87cd-2753bdb34546-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:05:23.744286 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.744253    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2da6f2f3-bfca-4327-bee7-9cb7c069a943" path="/var/lib/kubelet/pods/2da6f2f3-bfca-4327-bee7-9cb7c069a943/volumes"
Sep 18 16:05:23.943286 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.943255    2559 generic.go:356] "Generic (PLEG): container finished" podID="11942848-e367-48a3-b6be-13b1bc6a1db5" containerID="3684295bed0569bb5b2776cad2285699dc3ab476b9ff092d7d10ff3c0a8c7dbd" exitCode=0
Sep 18 16:05:23.943732 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.943296    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"
Sep 18 16:05:23.943732 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.943339    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86" event={"ID":"11942848-e367-48a3-b6be-13b1bc6a1db5","Type":"ContainerDied","Data":"3684295bed0569bb5b2776cad2285699dc3ab476b9ff092d7d10ff3c0a8c7dbd"}
Sep 18 16:05:23.943732 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.943392    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86" event={"ID":"11942848-e367-48a3-b6be-13b1bc6a1db5","Type":"ContainerDied","Data":"fe530c0d241e82a10451c42e9e518acb39e84ed68c20eb7861d5b112ddc98035"}
Sep 18 16:05:23.943732 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.943418    2559 scope.go:117] "RemoveContainer" containerID="3684295bed0569bb5b2776cad2285699dc3ab476b9ff092d7d10ff3c0a8c7dbd"
Sep 18 16:05:23.944765 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.944746    2559 generic.go:356] "Generic (PLEG): container finished" podID="65aa993d-9443-479c-87cd-2753bdb34546" containerID="cce21599ec1c272ae3358526fd6f8668b7b7d63671812ac8994cbff26b325e88" exitCode=0
Sep 18 16:05:23.944865 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.944799    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"
Sep 18 16:05:23.944865 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.944833    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8" event={"ID":"65aa993d-9443-479c-87cd-2753bdb34546","Type":"ContainerDied","Data":"cce21599ec1c272ae3358526fd6f8668b7b7d63671812ac8994cbff26b325e88"}
Sep 18 16:05:23.944976 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.944911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8" event={"ID":"65aa993d-9443-479c-87cd-2753bdb34546","Type":"ContainerDied","Data":"5bf604fbfebd560ad67ad77e7c3a6514476dddfaa34516883ba931a356d4a8c4"}
Sep 18 16:05:23.957340 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.957269    2559 scope.go:117] "RemoveContainer" containerID="3684295bed0569bb5b2776cad2285699dc3ab476b9ff092d7d10ff3c0a8c7dbd"
Sep 18 16:05:23.957660 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:05:23.957628    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3684295bed0569bb5b2776cad2285699dc3ab476b9ff092d7d10ff3c0a8c7dbd\": container with ID starting with 3684295bed0569bb5b2776cad2285699dc3ab476b9ff092d7d10ff3c0a8c7dbd not found: ID does not exist" containerID="3684295bed0569bb5b2776cad2285699dc3ab476b9ff092d7d10ff3c0a8c7dbd"
Sep 18 16:05:23.957728 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.957679    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3684295bed0569bb5b2776cad2285699dc3ab476b9ff092d7d10ff3c0a8c7dbd"} err="failed to get container status \"3684295bed0569bb5b2776cad2285699dc3ab476b9ff092d7d10ff3c0a8c7dbd\": rpc error: code = NotFound desc = could not find container \"3684295bed0569bb5b2776cad2285699dc3ab476b9ff092d7d10ff3c0a8c7dbd\": container with ID starting with 3684295bed0569bb5b2776cad2285699dc3ab476b9ff092d7d10ff3c0a8c7dbd not found: ID does not exist"
Sep 18 16:05:23.957728 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.957710    2559 scope.go:117] "RemoveContainer" containerID="cce21599ec1c272ae3358526fd6f8668b7b7d63671812ac8994cbff26b325e88"
Sep 18 16:05:23.964125 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.964104    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"]
Sep 18 16:05:23.967290 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.967270    2559 scope.go:117] "RemoveContainer" containerID="cce21599ec1c272ae3358526fd6f8668b7b7d63671812ac8994cbff26b325e88"
Sep 18 16:05:23.967591 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:05:23.967568    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce21599ec1c272ae3358526fd6f8668b7b7d63671812ac8994cbff26b325e88\": container with ID starting with cce21599ec1c272ae3358526fd6f8668b7b7d63671812ac8994cbff26b325e88 not found: ID does not exist" containerID="cce21599ec1c272ae3358526fd6f8668b7b7d63671812ac8994cbff26b325e88"
Sep 18 16:05:23.967671 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.967601    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce21599ec1c272ae3358526fd6f8668b7b7d63671812ac8994cbff26b325e88"} err="failed to get container status \"cce21599ec1c272ae3358526fd6f8668b7b7d63671812ac8994cbff26b325e88\": rpc error: code = NotFound desc = could not find container \"cce21599ec1c272ae3358526fd6f8668b7b7d63671812ac8994cbff26b325e88\": container with ID starting with cce21599ec1c272ae3358526fd6f8668b7b7d63671812ac8994cbff26b325e88 not found: ID does not exist"
Sep 18 16:05:23.969702 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.969682    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-90ebe5-5c674744df-88ks8"]
Sep 18 16:05:23.983147 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.983121    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"]
Sep 18 16:05:23.990802 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:23.990776    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-90ebe5-6956684666-77x86"]
Sep 18 16:05:24.061485 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:24.061425    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:24.634763 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:24.634725    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:25.440897 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:25.440840    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:25.743950 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:25.743851    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11942848-e367-48a3-b6be-13b1bc6a1db5" path="/var/lib/kubelet/pods/11942848-e367-48a3-b6be-13b1bc6a1db5/volumes"
Sep 18 16:05:25.744230 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:25.744218    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65aa993d-9443-479c-87cd-2753bdb34546" path="/var/lib/kubelet/pods/65aa993d-9443-479c-87cd-2753bdb34546/volumes"
Sep 18 16:05:26.197624 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:26.197588    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:28.739062 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:28.739026    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:32.442485 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:32.442451    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:32.543086 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:32.543057    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:33.699541 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:33.699505    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:05:42.545205 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:42.545162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:46.151742 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:46.151707    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:46.960390 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:46.960355    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:52.678334 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:52.678242    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:52.823139 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:52.823101    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:53.000497 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.000457    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:53.426645 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.426560    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm"]
Sep 18 16:05:53.427045 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.427027    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da6f2f3-bfca-4327-bee7-9cb7c069a943" containerName="revoke-keys"
Sep 18 16:05:53.427150 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.427047    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da6f2f3-bfca-4327-bee7-9cb7c069a943" containerName="revoke-keys"
Sep 18 16:05:53.427150 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.427075    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11942848-e367-48a3-b6be-13b1bc6a1db5" containerName="payload-pre-processing"
Sep 18 16:05:53.427150 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.427083    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="11942848-e367-48a3-b6be-13b1bc6a1db5" containerName="payload-pre-processing"
Sep 18 16:05:53.427150 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.427095    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65aa993d-9443-479c-87cd-2753bdb34546" containerName="payload-processing"
Sep 18 16:05:53.427150 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.427103    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="65aa993d-9443-479c-87cd-2753bdb34546" containerName="payload-processing"
Sep 18 16:05:53.427401 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.427204    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="65aa993d-9443-479c-87cd-2753bdb34546" containerName="payload-processing"
Sep 18 16:05:53.427401 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.427217    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da6f2f3-bfca-4327-bee7-9cb7c069a943" containerName="revoke-keys"
Sep 18 16:05:53.427401 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.427228    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="11942848-e367-48a3-b6be-13b1bc6a1db5" containerName="payload-pre-processing"
Sep 18 16:05:53.431876 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.431835    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm"
Sep 18 16:05:53.436291 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.436270    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 16:05:53.436633 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.436611    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:05:53.443610 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.443588    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm"]
Sep 18 16:05:53.578185 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.578151    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19066a5b-a792-47d3-83f3-f59aaa913f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm\" (UID: \"19066a5b-a792-47d3-83f3-f59aaa913f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm"
Sep 18 16:05:53.679595 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.679527    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19066a5b-a792-47d3-83f3-f59aaa913f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm\" (UID: \"19066a5b-a792-47d3-83f3-f59aaa913f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm"
Sep 18 16:05:53.680167 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.680145    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19066a5b-a792-47d3-83f3-f59aaa913f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm\" (UID: \"19066a5b-a792-47d3-83f3-f59aaa913f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm"
Sep 18 16:05:53.741834 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:53.741805    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm"
Sep 18 16:05:54.011650 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:54.011622    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm"]
Sep 18 16:05:54.012981 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:05:54.012952    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19066a5b_a792_47d3_83f3_f59aaa913f43.slice/crio-86fab1f884441851622ab3b5a532575fa3d04fdc6c4cd49638c0cda877450360 WatchSource:0}: Error finding container 86fab1f884441851622ab3b5a532575fa3d04fdc6c4cd49638c0cda877450360: Status 404 returned error can't find the container with id 86fab1f884441851622ab3b5a532575fa3d04fdc6c4cd49638c0cda877450360
Sep 18 16:05:54.100715 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:54.100672    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm" event={"ID":"19066a5b-a792-47d3-83f3-f59aaa913f43","Type":"ContainerStarted","Data":"86fab1f884441851622ab3b5a532575fa3d04fdc6c4cd49638c0cda877450360"}
Sep 18 16:05:54.198301 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:54.198274    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:54.983297 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:54.983256    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:55.107400 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:55.107369    2559 generic.go:356] "Generic (PLEG): container finished" podID="19066a5b-a792-47d3-83f3-f59aaa913f43" containerID="70e2954834f396a6b06b60655f8fd7474438a937ab8fd9b1b711191934d8cff5" exitCode=0
Sep 18 16:05:55.107563 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:55.107431    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm" event={"ID":"19066a5b-a792-47d3-83f3-f59aaa913f43","Type":"ContainerDied","Data":"70e2954834f396a6b06b60655f8fd7474438a937ab8fd9b1b711191934d8cff5"}
Sep 18 16:05:55.815665 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:55.815623    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:56.240918 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:56.240893    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm"
Sep 18 16:05:56.408052 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:56.407976    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19066a5b-a792-47d3-83f3-f59aaa913f43-maas-api-service-ca\") pod \"19066a5b-a792-47d3-83f3-f59aaa913f43\" (UID: \"19066a5b-a792-47d3-83f3-f59aaa913f43\") "
Sep 18 16:05:56.408341 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:56.408318    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19066a5b-a792-47d3-83f3-f59aaa913f43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "19066a5b-a792-47d3-83f3-f59aaa913f43" (UID: "19066a5b-a792-47d3-83f3-f59aaa913f43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:05:56.509211 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:56.509182    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19066a5b-a792-47d3-83f3-f59aaa913f43-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:05:56.568288 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:56.568255    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:05:57.119429 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:57.119398    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm"
Sep 18 16:05:57.119429 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:57.119403    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm" event={"ID":"19066a5b-a792-47d3-83f3-f59aaa913f43","Type":"ContainerDied","Data":"86fab1f884441851622ab3b5a532575fa3d04fdc6c4cd49638c0cda877450360"}
Sep 18 16:05:57.119642 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:05:57.119483    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86fab1f884441851622ab3b5a532575fa3d04fdc6c4cd49638c0cda877450360"
Sep 18 16:06:04.138604 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.138564    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:04.166916 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.166878    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"]
Sep 18 16:06:04.167197 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.167167    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s" podUID="05a44e79-492b-4246-9580-e89ce8be7b7b" containerName="payload-processing" containerID="cri-o://ae75e18badcc50499d6de42dd67db58afecdc0138834f73ee1e4dd89414d93af" gracePeriod=30
Sep 18 16:06:04.177226 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.177196    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"]
Sep 18 16:06:04.177464 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.177422    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw" podUID="e82740fe-d8fb-4276-943a-660e36ad05af" containerName="payload-pre-processing" containerID="cri-o://03f4f5c034602985e3f0a56071041f864084ebcf2fc631d36b7029b2467e2aab" gracePeriod=30
Sep 18 16:06:04.239793 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.239762    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:04.341224 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.340746    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:04.390511 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.390435    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm"]
Sep 18 16:06:04.397326 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.397300    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e41875-l8vkm"]
Sep 18 16:06:04.421636 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.421609    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:04.423757 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.423741    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"
Sep 18 16:06:04.456483 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.456461    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"
Sep 18 16:06:04.577723 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.577693    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slwtw\" (UniqueName: \"kubernetes.io/projected/e82740fe-d8fb-4276-943a-660e36ad05af-kube-api-access-slwtw\") pod \"e82740fe-d8fb-4276-943a-660e36ad05af\" (UID: \"e82740fe-d8fb-4276-943a-660e36ad05af\") "
Sep 18 16:06:04.577916 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.577774    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05a44e79-492b-4246-9580-e89ce8be7b7b-plugins-config-volume\") pod \"05a44e79-492b-4246-9580-e89ce8be7b7b\" (UID: \"05a44e79-492b-4246-9580-e89ce8be7b7b\") "
Sep 18 16:06:04.577965 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.577946    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj7b9\" (UniqueName: \"kubernetes.io/projected/05a44e79-492b-4246-9580-e89ce8be7b7b-kube-api-access-gj7b9\") pod \"05a44e79-492b-4246-9580-e89ce8be7b7b\" (UID: \"05a44e79-492b-4246-9580-e89ce8be7b7b\") "
Sep 18 16:06:04.578052 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.578036    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82740fe-d8fb-4276-943a-660e36ad05af-plugins-config-volume\") pod \"e82740fe-d8fb-4276-943a-660e36ad05af\" (UID: \"e82740fe-d8fb-4276-943a-660e36ad05af\") "
Sep 18 16:06:04.578141 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.578122    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a44e79-492b-4246-9580-e89ce8be7b7b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05a44e79-492b-4246-9580-e89ce8be7b7b" (UID: "05a44e79-492b-4246-9580-e89ce8be7b7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:06:04.578360 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.578342    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05a44e79-492b-4246-9580-e89ce8be7b7b-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:04.578460 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.578404    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82740fe-d8fb-4276-943a-660e36ad05af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e82740fe-d8fb-4276-943a-660e36ad05af" (UID: "e82740fe-d8fb-4276-943a-660e36ad05af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:06:04.579749 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.579734    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82740fe-d8fb-4276-943a-660e36ad05af-kube-api-access-slwtw" (OuterVolumeSpecName: "kube-api-access-slwtw") pod "e82740fe-d8fb-4276-943a-660e36ad05af" (UID: "e82740fe-d8fb-4276-943a-660e36ad05af"). InnerVolumeSpecName "kube-api-access-slwtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:06:04.579825 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.579807    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a44e79-492b-4246-9580-e89ce8be7b7b-kube-api-access-gj7b9" (OuterVolumeSpecName: "kube-api-access-gj7b9") pod "05a44e79-492b-4246-9580-e89ce8be7b7b" (UID: "05a44e79-492b-4246-9580-e89ce8be7b7b"). InnerVolumeSpecName "kube-api-access-gj7b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:06:04.679649 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.679574    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slwtw\" (UniqueName: \"kubernetes.io/projected/e82740fe-d8fb-4276-943a-660e36ad05af-kube-api-access-slwtw\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:04.679649 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.679600    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj7b9\" (UniqueName: \"kubernetes.io/projected/05a44e79-492b-4246-9580-e89ce8be7b7b-kube-api-access-gj7b9\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:04.679649 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.679609    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82740fe-d8fb-4276-943a-660e36ad05af-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:04.697162 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.697130    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:04.824302 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:04.824274    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:05.160165 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.160131    2559 generic.go:356] "Generic (PLEG): container finished" podID="05a44e79-492b-4246-9580-e89ce8be7b7b" containerID="ae75e18badcc50499d6de42dd67db58afecdc0138834f73ee1e4dd89414d93af" exitCode=0
Sep 18 16:06:05.160631 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.160177    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"
Sep 18 16:06:05.160631 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.160205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s" event={"ID":"05a44e79-492b-4246-9580-e89ce8be7b7b","Type":"ContainerDied","Data":"ae75e18badcc50499d6de42dd67db58afecdc0138834f73ee1e4dd89414d93af"}
Sep 18 16:06:05.160631 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.160257    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s" event={"ID":"05a44e79-492b-4246-9580-e89ce8be7b7b","Type":"ContainerDied","Data":"3a5c731ce416cb50b70b5f1e401fc66921c9f62f8c2c758c901dc2b12f9281b9"}
Sep 18 16:06:05.160631 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.160284    2559 scope.go:117] "RemoveContainer" containerID="ae75e18badcc50499d6de42dd67db58afecdc0138834f73ee1e4dd89414d93af"
Sep 18 16:06:05.161720 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.161699    2559 generic.go:356] "Generic (PLEG): container finished" podID="e82740fe-d8fb-4276-943a-660e36ad05af" containerID="03f4f5c034602985e3f0a56071041f864084ebcf2fc631d36b7029b2467e2aab" exitCode=0
Sep 18 16:06:05.161807 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.161739    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"
Sep 18 16:06:05.161807 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.161780    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw" event={"ID":"e82740fe-d8fb-4276-943a-660e36ad05af","Type":"ContainerDied","Data":"03f4f5c034602985e3f0a56071041f864084ebcf2fc631d36b7029b2467e2aab"}
Sep 18 16:06:05.161807 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.161824    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw" event={"ID":"e82740fe-d8fb-4276-943a-660e36ad05af","Type":"ContainerDied","Data":"60b217f50a8f80d134072ed11a75f68afbf85fe1d30ae8eddbf61ca1d9909c29"}
Sep 18 16:06:05.169590 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.169374    2559 scope.go:117] "RemoveContainer" containerID="ae75e18badcc50499d6de42dd67db58afecdc0138834f73ee1e4dd89414d93af"
Sep 18 16:06:05.169655 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:06:05.169638    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae75e18badcc50499d6de42dd67db58afecdc0138834f73ee1e4dd89414d93af\": container with ID starting with ae75e18badcc50499d6de42dd67db58afecdc0138834f73ee1e4dd89414d93af not found: ID does not exist" containerID="ae75e18badcc50499d6de42dd67db58afecdc0138834f73ee1e4dd89414d93af"
Sep 18 16:06:05.169708 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.169663    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae75e18badcc50499d6de42dd67db58afecdc0138834f73ee1e4dd89414d93af"} err="failed to get container status \"ae75e18badcc50499d6de42dd67db58afecdc0138834f73ee1e4dd89414d93af\": rpc error: code = NotFound desc = could not find container \"ae75e18badcc50499d6de42dd67db58afecdc0138834f73ee1e4dd89414d93af\": container with ID starting with ae75e18badcc50499d6de42dd67db58afecdc0138834f73ee1e4dd89414d93af not found: ID does not exist"
Sep 18 16:06:05.169708 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.169680    2559 scope.go:117] "RemoveContainer" containerID="03f4f5c034602985e3f0a56071041f864084ebcf2fc631d36b7029b2467e2aab"
Sep 18 16:06:05.180344 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.180324    2559 scope.go:117] "RemoveContainer" containerID="03f4f5c034602985e3f0a56071041f864084ebcf2fc631d36b7029b2467e2aab"
Sep 18 16:06:05.180638 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:06:05.180618    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f4f5c034602985e3f0a56071041f864084ebcf2fc631d36b7029b2467e2aab\": container with ID starting with 03f4f5c034602985e3f0a56071041f864084ebcf2fc631d36b7029b2467e2aab not found: ID does not exist" containerID="03f4f5c034602985e3f0a56071041f864084ebcf2fc631d36b7029b2467e2aab"
Sep 18 16:06:05.180701 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.180648    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f4f5c034602985e3f0a56071041f864084ebcf2fc631d36b7029b2467e2aab"} err="failed to get container status \"03f4f5c034602985e3f0a56071041f864084ebcf2fc631d36b7029b2467e2aab\": rpc error: code = NotFound desc = could not find container \"03f4f5c034602985e3f0a56071041f864084ebcf2fc631d36b7029b2467e2aab\": container with ID starting with 03f4f5c034602985e3f0a56071041f864084ebcf2fc631d36b7029b2467e2aab not found: ID does not exist"
Sep 18 16:06:05.188307 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.188284    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"]
Sep 18 16:06:05.196741 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.196715    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e41875-64898f859f-7sd7s"]
Sep 18 16:06:05.212009 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.211988    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"]
Sep 18 16:06:05.215589 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.215572    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e41875-7b6d844c4-9nqlw"]
Sep 18 16:06:05.480935 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.480902    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:05.744093 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.744019    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05a44e79-492b-4246-9580-e89ce8be7b7b" path="/var/lib/kubelet/pods/05a44e79-492b-4246-9580-e89ce8be7b7b/volumes"
Sep 18 16:06:05.744346 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.744334    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19066a5b-a792-47d3-83f3-f59aaa913f43" path="/var/lib/kubelet/pods/19066a5b-a792-47d3-83f3-f59aaa913f43/volumes"
Sep 18 16:06:05.744645 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:05.744634    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e82740fe-d8fb-4276-943a-660e36ad05af" path="/var/lib/kubelet/pods/e82740fe-d8fb-4276-943a-660e36ad05af/volumes"
Sep 18 16:06:06.221886 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:06.221822    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:07.283104 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:07.283074    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:06:10.892241 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:10.892203    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:10.972346 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:10.972309    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:11.177537 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.177452    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:11.502767 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.502736    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq"]
Sep 18 16:06:11.503190 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.503176    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a44e79-492b-4246-9580-e89ce8be7b7b" containerName="payload-processing"
Sep 18 16:06:11.503236 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.503191    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a44e79-492b-4246-9580-e89ce8be7b7b" containerName="payload-processing"
Sep 18 16:06:11.503236 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.503199    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e82740fe-d8fb-4276-943a-660e36ad05af" containerName="payload-pre-processing"
Sep 18 16:06:11.503236 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.503204    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e82740fe-d8fb-4276-943a-660e36ad05af" containerName="payload-pre-processing"
Sep 18 16:06:11.503236 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.503214    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19066a5b-a792-47d3-83f3-f59aaa913f43" containerName="revoke-keys"
Sep 18 16:06:11.503236 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.503219    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="19066a5b-a792-47d3-83f3-f59aaa913f43" containerName="revoke-keys"
Sep 18 16:06:11.503390 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.503280    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e82740fe-d8fb-4276-943a-660e36ad05af" containerName="payload-pre-processing"
Sep 18 16:06:11.503390 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.503290    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="05a44e79-492b-4246-9580-e89ce8be7b7b" containerName="payload-processing"
Sep 18 16:06:11.503390 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.503297    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="19066a5b-a792-47d3-83f3-f59aaa913f43" containerName="revoke-keys"
Sep 18 16:06:11.507361 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.507345    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq"
Sep 18 16:06:11.510537 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.510520    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:06:11.510627 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.510521    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 16:06:11.515258 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.515236    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq"]
Sep 18 16:06:11.634645 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.634602    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dc1d4b-70a4-4029-a683-5dd9716b4dee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq\" (UID: \"b0dc1d4b-70a4-4029-a683-5dd9716b4dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq"
Sep 18 16:06:11.735086 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.735053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dc1d4b-70a4-4029-a683-5dd9716b4dee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq\" (UID: \"b0dc1d4b-70a4-4029-a683-5dd9716b4dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq"
Sep 18 16:06:11.735666 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.735645    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dc1d4b-70a4-4029-a683-5dd9716b4dee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq\" (UID: \"b0dc1d4b-70a4-4029-a683-5dd9716b4dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq"
Sep 18 16:06:11.811902 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.811810    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:11.818731 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:11.818715    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq"
Sep 18 16:06:12.093089 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:12.093057    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq"]
Sep 18 16:06:12.094746 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:06:12.094716    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0dc1d4b_70a4_4029_a683_5dd9716b4dee.slice/crio-35f879916836f6ae895b5fb050d944ab7ff77c35abcba187d85e70225ed7034a WatchSource:0}: Error finding container 35f879916836f6ae895b5fb050d944ab7ff77c35abcba187d85e70225ed7034a: Status 404 returned error can't find the container with id 35f879916836f6ae895b5fb050d944ab7ff77c35abcba187d85e70225ed7034a
Sep 18 16:06:12.199915 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:12.199885    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq" event={"ID":"b0dc1d4b-70a4-4029-a683-5dd9716b4dee","Type":"ContainerStarted","Data":"35f879916836f6ae895b5fb050d944ab7ff77c35abcba187d85e70225ed7034a"}
Sep 18 16:06:12.643615 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:12.643535    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:13.207406 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:13.207371    2559 generic.go:356] "Generic (PLEG): container finished" podID="b0dc1d4b-70a4-4029-a683-5dd9716b4dee" containerID="7085d3882def6c68c4ffc67498b3d0aeeb5102b9ff8fc86c1ee40a85c43d4740" exitCode=0
Sep 18 16:06:13.207771 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:13.207440    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq" event={"ID":"b0dc1d4b-70a4-4029-a683-5dd9716b4dee","Type":"ContainerDied","Data":"7085d3882def6c68c4ffc67498b3d0aeeb5102b9ff8fc86c1ee40a85c43d4740"}
Sep 18 16:06:13.318667 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:13.318635    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:14.214092 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:14.214062    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:14.337182 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:14.337160    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq"
Sep 18 16:06:14.354399 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:14.354365    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dc1d4b-70a4-4029-a683-5dd9716b4dee-maas-api-service-ca\") pod \"b0dc1d4b-70a4-4029-a683-5dd9716b4dee\" (UID: \"b0dc1d4b-70a4-4029-a683-5dd9716b4dee\") "
Sep 18 16:06:14.354920 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:14.354882    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0dc1d4b-70a4-4029-a683-5dd9716b4dee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0dc1d4b-70a4-4029-a683-5dd9716b4dee" (UID: "b0dc1d4b-70a4-4029-a683-5dd9716b4dee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:06:14.455908 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:14.455876    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dc1d4b-70a4-4029-a683-5dd9716b4dee-maas-api-service-ca\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:15.219507 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:15.219472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq" event={"ID":"b0dc1d4b-70a4-4029-a683-5dd9716b4dee","Type":"ContainerDied","Data":"35f879916836f6ae895b5fb050d944ab7ff77c35abcba187d85e70225ed7034a"}
Sep 18 16:06:15.219507 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:15.219499    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq"
Sep 18 16:06:15.219976 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:15.219528    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35f879916836f6ae895b5fb050d944ab7ff77c35abcba187d85e70225ed7034a"
Sep 18 16:06:22.041414 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.041369    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:22.071063 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.071031    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"]
Sep 18 16:06:22.071292 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.071265    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb" podUID="82afb053-d882-4da8-8544-bea819831570" containerName="payload-processing" containerID="cri-o://f467da982fa651a91ee460a1b6f5db9d0b5d9298a32210ddae53df8f9a8f587b" gracePeriod=30
Sep 18 16:06:22.080115 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.080091    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"]
Sep 18 16:06:22.080318 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.080277    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft" podUID="b261b9f5-63ab-459e-894d-d8d9898c95f0" containerName="payload-pre-processing" containerID="cri-o://4d6763cd444290cd4076bfcc6f5d4159a00eb3acd12bbf044b78b0b636e54e32" gracePeriod=30
Sep 18 16:06:22.118615 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.118580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:22.189276 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.189240    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:22.262208 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.262153    2559 generic.go:356] "Generic (PLEG): container finished" podID="82afb053-d882-4da8-8544-bea819831570" containerID="f467da982fa651a91ee460a1b6f5db9d0b5d9298a32210ddae53df8f9a8f587b" exitCode=0
Sep 18 16:06:22.262208 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.262177    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb" event={"ID":"82afb053-d882-4da8-8544-bea819831570","Type":"ContainerDied","Data":"f467da982fa651a91ee460a1b6f5db9d0b5d9298a32210ddae53df8f9a8f587b"}
Sep 18 16:06:22.264036 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.263967    2559 generic.go:356] "Generic (PLEG): container finished" podID="b261b9f5-63ab-459e-894d-d8d9898c95f0" containerID="4d6763cd444290cd4076bfcc6f5d4159a00eb3acd12bbf044b78b0b636e54e32" exitCode=0
Sep 18 16:06:22.264172 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.264031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft" event={"ID":"b261b9f5-63ab-459e-894d-d8d9898c95f0","Type":"ContainerDied","Data":"4d6763cd444290cd4076bfcc6f5d4159a00eb3acd12bbf044b78b0b636e54e32"}
Sep 18 16:06:22.294682 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.294621    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq"]
Sep 18 16:06:22.297724 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.297701    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f14aba-wqvqq"]
Sep 18 16:06:22.311148 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.311105    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:22.330045 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.330019    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"
Sep 18 16:06:22.359180 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.359156    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"
Sep 18 16:06:22.429254 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.429225    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2vn4\" (UniqueName: \"kubernetes.io/projected/b261b9f5-63ab-459e-894d-d8d9898c95f0-kube-api-access-b2vn4\") pod \"b261b9f5-63ab-459e-894d-d8d9898c95f0\" (UID: \"b261b9f5-63ab-459e-894d-d8d9898c95f0\") "
Sep 18 16:06:22.429414 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.429307    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b261b9f5-63ab-459e-894d-d8d9898c95f0-plugins-config-volume\") pod \"b261b9f5-63ab-459e-894d-d8d9898c95f0\" (UID: \"b261b9f5-63ab-459e-894d-d8d9898c95f0\") "
Sep 18 16:06:22.429414 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.429324    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82afb053-d882-4da8-8544-bea819831570-plugins-config-volume\") pod \"82afb053-d882-4da8-8544-bea819831570\" (UID: \"82afb053-d882-4da8-8544-bea819831570\") "
Sep 18 16:06:22.429414 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.429345    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpjmg\" (UniqueName: \"kubernetes.io/projected/82afb053-d882-4da8-8544-bea819831570-kube-api-access-mpjmg\") pod \"82afb053-d882-4da8-8544-bea819831570\" (UID: \"82afb053-d882-4da8-8544-bea819831570\") "
Sep 18 16:06:22.429679 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.429659    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82afb053-d882-4da8-8544-bea819831570-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82afb053-d882-4da8-8544-bea819831570" (UID: "82afb053-d882-4da8-8544-bea819831570"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:06:22.429741 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.429716    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b261b9f5-63ab-459e-894d-d8d9898c95f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b261b9f5-63ab-459e-894d-d8d9898c95f0" (UID: "b261b9f5-63ab-459e-894d-d8d9898c95f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:06:22.431311 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.431284    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b261b9f5-63ab-459e-894d-d8d9898c95f0-kube-api-access-b2vn4" (OuterVolumeSpecName: "kube-api-access-b2vn4") pod "b261b9f5-63ab-459e-894d-d8d9898c95f0" (UID: "b261b9f5-63ab-459e-894d-d8d9898c95f0"). InnerVolumeSpecName "kube-api-access-b2vn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:06:22.431405 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.431388    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82afb053-d882-4da8-8544-bea819831570-kube-api-access-mpjmg" (OuterVolumeSpecName: "kube-api-access-mpjmg") pod "82afb053-d882-4da8-8544-bea819831570" (UID: "82afb053-d882-4da8-8544-bea819831570"). InnerVolumeSpecName "kube-api-access-mpjmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:06:22.473708 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.473679    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:22.530426 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.530402    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b261b9f5-63ab-459e-894d-d8d9898c95f0-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:22.530426 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.530424    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82afb053-d882-4da8-8544-bea819831570-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:22.530555 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.530434    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpjmg\" (UniqueName: \"kubernetes.io/projected/82afb053-d882-4da8-8544-bea819831570-kube-api-access-mpjmg\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:22.530555 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.530443    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2vn4\" (UniqueName: \"kubernetes.io/projected/b261b9f5-63ab-459e-894d-d8d9898c95f0-kube-api-access-b2vn4\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:22.705443 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.705414    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:22.806179 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:22.806141    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:23.271387 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.271354    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"
Sep 18 16:06:23.271788 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.271391    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft" event={"ID":"b261b9f5-63ab-459e-894d-d8d9898c95f0","Type":"ContainerDied","Data":"73909df8d81ef77b3df3b003723e8d1764aaa7265edce5be683ffa9b497162e8"}
Sep 18 16:06:23.271788 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.271447    2559 scope.go:117] "RemoveContainer" containerID="4d6763cd444290cd4076bfcc6f5d4159a00eb3acd12bbf044b78b0b636e54e32"
Sep 18 16:06:23.272868 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.272840    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"
Sep 18 16:06:23.272953 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.272840    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb" event={"ID":"82afb053-d882-4da8-8544-bea819831570","Type":"ContainerDied","Data":"fb97185dd7cbc6ba4f9008975a631c641bd98fb7607721bcec867d26bfaf2bdf"}
Sep 18 16:06:23.280343 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.280177    2559 scope.go:117] "RemoveContainer" containerID="f467da982fa651a91ee460a1b6f5db9d0b5d9298a32210ddae53df8f9a8f587b"
Sep 18 16:06:23.318892 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.318848    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"]
Sep 18 16:06:23.328710 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.328681    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f14aba-588477f867-ds8rb"]
Sep 18 16:06:23.369118 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.369054    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"]
Sep 18 16:06:23.378969 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.378943    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f14aba-768db85d-jpdft"]
Sep 18 16:06:23.515702 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.515672    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:23.743304 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.743274    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82afb053-d882-4da8-8544-bea819831570" path="/var/lib/kubelet/pods/82afb053-d882-4da8-8544-bea819831570/volumes"
Sep 18 16:06:23.743595 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.743582    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0dc1d4b-70a4-4029-a683-5dd9716b4dee" path="/var/lib/kubelet/pods/b0dc1d4b-70a4-4029-a683-5dd9716b4dee/volumes"
Sep 18 16:06:23.743906 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:23.743893    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b261b9f5-63ab-459e-894d-d8d9898c95f0" path="/var/lib/kubelet/pods/b261b9f5-63ab-459e-894d-d8d9898c95f0/volumes"
Sep 18 16:06:24.977538 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:24.977505    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:06:28.656087 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:28.656049    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:28.754783 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:28.754745    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:28.956967 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:28.956937    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:30.075900 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:30.075866    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:30.657746 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:30.657715    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:39.866555 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:39.866522    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj"]
Sep 18 16:06:39.868587 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:39.868545    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:39.870228 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:39.870202    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4e81aa-29829120-6mrgj"]
Sep 18 16:06:39.898716 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:39.898591    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"]
Sep 18 16:06:39.899052 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:39.899001    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m" podUID="199bef39-11ee-4aca-bd23-778459f6b5e0" containerName="payload-processing" containerID="cri-o://456a6f5ca48024743a2895eb9a5cb242590dec0d308dd365bdd3d0e1e24c229f" gracePeriod=30
Sep 18 16:06:39.904959 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:39.904934    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"]
Sep 18 16:06:39.905170 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:39.905147    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx" podUID="fca1c4b5-d927-4570-a16c-93de369c34f2" containerName="payload-pre-processing" containerID="cri-o://b996dbc34f622d70f852e6a225387eeab0175407b966e3f99d6c0c87ee074656" gracePeriod=30
Sep 18 16:06:39.949296 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:39.949269    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:40.065317 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.064706    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:40.157430 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.157400    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"
Sep 18 16:06:40.166695 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.166625    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:40.181608 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.181577    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"
Sep 18 16:06:40.280723 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.280690    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvbrg\" (UniqueName: \"kubernetes.io/projected/199bef39-11ee-4aca-bd23-778459f6b5e0-kube-api-access-tvbrg\") pod \"199bef39-11ee-4aca-bd23-778459f6b5e0\" (UID: \"199bef39-11ee-4aca-bd23-778459f6b5e0\") "
Sep 18 16:06:40.280935 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.280734    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qggz\" (UniqueName: \"kubernetes.io/projected/fca1c4b5-d927-4570-a16c-93de369c34f2-kube-api-access-4qggz\") pod \"fca1c4b5-d927-4570-a16c-93de369c34f2\" (UID: \"fca1c4b5-d927-4570-a16c-93de369c34f2\") "
Sep 18 16:06:40.280935 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.280769    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199bef39-11ee-4aca-bd23-778459f6b5e0-plugins-config-volume\") pod \"199bef39-11ee-4aca-bd23-778459f6b5e0\" (UID: \"199bef39-11ee-4aca-bd23-778459f6b5e0\") "
Sep 18 16:06:40.280935 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.280822    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca1c4b5-d927-4570-a16c-93de369c34f2-plugins-config-volume\") pod \"fca1c4b5-d927-4570-a16c-93de369c34f2\" (UID: \"fca1c4b5-d927-4570-a16c-93de369c34f2\") "
Sep 18 16:06:40.281203 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.281168    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca1c4b5-d927-4570-a16c-93de369c34f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fca1c4b5-d927-4570-a16c-93de369c34f2" (UID: "fca1c4b5-d927-4570-a16c-93de369c34f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:06:40.281203 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.281192    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199bef39-11ee-4aca-bd23-778459f6b5e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "199bef39-11ee-4aca-bd23-778459f6b5e0" (UID: "199bef39-11ee-4aca-bd23-778459f6b5e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:06:40.282792 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.282770    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199bef39-11ee-4aca-bd23-778459f6b5e0-kube-api-access-tvbrg" (OuterVolumeSpecName: "kube-api-access-tvbrg") pod "199bef39-11ee-4aca-bd23-778459f6b5e0" (UID: "199bef39-11ee-4aca-bd23-778459f6b5e0"). InnerVolumeSpecName "kube-api-access-tvbrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:06:40.282867 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.282796    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca1c4b5-d927-4570-a16c-93de369c34f2-kube-api-access-4qggz" (OuterVolumeSpecName: "kube-api-access-4qggz") pod "fca1c4b5-d927-4570-a16c-93de369c34f2" (UID: "fca1c4b5-d927-4570-a16c-93de369c34f2"). InnerVolumeSpecName "kube-api-access-4qggz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:06:40.361338 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.361304    2559 generic.go:356] "Generic (PLEG): container finished" podID="199bef39-11ee-4aca-bd23-778459f6b5e0" containerID="456a6f5ca48024743a2895eb9a5cb242590dec0d308dd365bdd3d0e1e24c229f" exitCode=0
Sep 18 16:06:40.361512 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.361353    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"
Sep 18 16:06:40.361512 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.361392    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m" event={"ID":"199bef39-11ee-4aca-bd23-778459f6b5e0","Type":"ContainerDied","Data":"456a6f5ca48024743a2895eb9a5cb242590dec0d308dd365bdd3d0e1e24c229f"}
Sep 18 16:06:40.361512 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.361456    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m" event={"ID":"199bef39-11ee-4aca-bd23-778459f6b5e0","Type":"ContainerDied","Data":"3bb8a555bc47422a71592e33ca176345552ff63e78100c79e3d5f577426cbd6e"}
Sep 18 16:06:40.361512 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.361488    2559 scope.go:117] "RemoveContainer" containerID="456a6f5ca48024743a2895eb9a5cb242590dec0d308dd365bdd3d0e1e24c229f"
Sep 18 16:06:40.362992 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.362968    2559 generic.go:356] "Generic (PLEG): container finished" podID="fca1c4b5-d927-4570-a16c-93de369c34f2" containerID="b996dbc34f622d70f852e6a225387eeab0175407b966e3f99d6c0c87ee074656" exitCode=0
Sep 18 16:06:40.363104 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.363016    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"
Sep 18 16:06:40.363104 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.363062    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx" event={"ID":"fca1c4b5-d927-4570-a16c-93de369c34f2","Type":"ContainerDied","Data":"b996dbc34f622d70f852e6a225387eeab0175407b966e3f99d6c0c87ee074656"}
Sep 18 16:06:40.363104 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.363101    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx" event={"ID":"fca1c4b5-d927-4570-a16c-93de369c34f2","Type":"ContainerDied","Data":"85fc318d36596df137368a466dd446945659f13833846c04339842a8fd9bbed4"}
Sep 18 16:06:40.370400 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.370381    2559 scope.go:117] "RemoveContainer" containerID="456a6f5ca48024743a2895eb9a5cb242590dec0d308dd365bdd3d0e1e24c229f"
Sep 18 16:06:40.370688 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:06:40.370665    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456a6f5ca48024743a2895eb9a5cb242590dec0d308dd365bdd3d0e1e24c229f\": container with ID starting with 456a6f5ca48024743a2895eb9a5cb242590dec0d308dd365bdd3d0e1e24c229f not found: ID does not exist" containerID="456a6f5ca48024743a2895eb9a5cb242590dec0d308dd365bdd3d0e1e24c229f"
Sep 18 16:06:40.370806 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.370694    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456a6f5ca48024743a2895eb9a5cb242590dec0d308dd365bdd3d0e1e24c229f"} err="failed to get container status \"456a6f5ca48024743a2895eb9a5cb242590dec0d308dd365bdd3d0e1e24c229f\": rpc error: code = NotFound desc = could not find container \"456a6f5ca48024743a2895eb9a5cb242590dec0d308dd365bdd3d0e1e24c229f\": container with ID starting with 456a6f5ca48024743a2895eb9a5cb242590dec0d308dd365bdd3d0e1e24c229f not found: ID does not exist"
Sep 18 16:06:40.370806 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.370715    2559 scope.go:117] "RemoveContainer" containerID="b996dbc34f622d70f852e6a225387eeab0175407b966e3f99d6c0c87ee074656"
Sep 18 16:06:40.378766 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.378748    2559 scope.go:117] "RemoveContainer" containerID="b996dbc34f622d70f852e6a225387eeab0175407b966e3f99d6c0c87ee074656"
Sep 18 16:06:40.379054 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:06:40.379037    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b996dbc34f622d70f852e6a225387eeab0175407b966e3f99d6c0c87ee074656\": container with ID starting with b996dbc34f622d70f852e6a225387eeab0175407b966e3f99d6c0c87ee074656 not found: ID does not exist" containerID="b996dbc34f622d70f852e6a225387eeab0175407b966e3f99d6c0c87ee074656"
Sep 18 16:06:40.379116 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.379059    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b996dbc34f622d70f852e6a225387eeab0175407b966e3f99d6c0c87ee074656"} err="failed to get container status \"b996dbc34f622d70f852e6a225387eeab0175407b966e3f99d6c0c87ee074656\": rpc error: code = NotFound desc = could not find container \"b996dbc34f622d70f852e6a225387eeab0175407b966e3f99d6c0c87ee074656\": container with ID starting with b996dbc34f622d70f852e6a225387eeab0175407b966e3f99d6c0c87ee074656 not found: ID does not exist"
Sep 18 16:06:40.382442 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.382425    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvbrg\" (UniqueName: \"kubernetes.io/projected/199bef39-11ee-4aca-bd23-778459f6b5e0-kube-api-access-tvbrg\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:40.382499 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.382444    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qggz\" (UniqueName: \"kubernetes.io/projected/fca1c4b5-d927-4570-a16c-93de369c34f2-kube-api-access-4qggz\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:40.382499 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.382455    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199bef39-11ee-4aca-bd23-778459f6b5e0-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:40.382499 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.382464    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca1c4b5-d927-4570-a16c-93de369c34f2-plugins-config-volume\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:40.388911 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.388848    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"]
Sep 18 16:06:40.395435 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.395415    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4e81aa-7b84b5c6f5-nmn9m"]
Sep 18 16:06:40.410528 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.410504    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"]
Sep 18 16:06:40.416162 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.416143    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4e81aa-9587bd5f8-rcnkx"]
Sep 18 16:06:40.504251 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.504217    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:40.607134 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.607097    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:40.843388 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.843357    2559 scope.go:117] "RemoveContainer" containerID="26d13209c2d84146dd9feace54a5f5aa789ae0f673b3ce2191ef2829e455ea16"
Sep 18 16:06:40.851543 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.851524    2559 scope.go:117] "RemoveContainer" containerID="12bd710fd93be75a35178ffd889fe9bf09c34d0ec667bac92b90731073f515d3"
Sep 18 16:06:40.858892 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.858871    2559 scope.go:117] "RemoveContainer" containerID="7bbe913a1659dcf5cdbfd9ad2ace80b4226e05b1c326419bec18a44782d4454c"
Sep 18 16:06:40.866827 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:40.866797    2559 scope.go:117] "RemoveContainer" containerID="1221c1d382e9ee0cb0034faa524204389c427f69264d9f30cbeb672cd3101e2d"
Sep 18 16:06:41.748962 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:41.748925    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199bef39-11ee-4aca-bd23-778459f6b5e0" path="/var/lib/kubelet/pods/199bef39-11ee-4aca-bd23-778459f6b5e0/volumes"
Sep 18 16:06:41.749208 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:41.749195    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ff73ef8-4a05-4222-9133-f388bffff7f9" path="/var/lib/kubelet/pods/4ff73ef8-4a05-4222-9133-f388bffff7f9/volumes"
Sep 18 16:06:41.749474 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:41.749462    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fca1c4b5-d927-4570-a16c-93de369c34f2" path="/var/lib/kubelet/pods/fca1c4b5-d927-4570-a16c-93de369c34f2/volumes"
Sep 18 16:06:42.673432 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:42.673399    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:06:46.407369 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:46.407317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:46.507499 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:46.507464    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:46.620717 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:46.620683    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"]
Sep 18 16:06:46.620967 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:46.620943    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerName="istio-proxy" containerID="cri-o://b792b8d3e9bf44350a7c12f5f51df2affba39cb7dcd628d2def381831918a4db" gracePeriod=30
Sep 18 16:06:46.639667 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:46.639633    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:06:50.571905 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:50.571868    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:06:51.857703 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.857679    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 16:06:51.978156 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978126    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-data\") pod \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") "
Sep 18 16:06:51.978338 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978162    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istiod-ca-cert\") pod \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") "
Sep 18 16:06:51.978338 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978198    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-credential-socket\") pod \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") "
Sep 18 16:06:51.978338 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978222    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-workload-certs\") pod \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") "
Sep 18 16:06:51.978338 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978237    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-envoy\") pod \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") "
Sep 18 16:06:51.978338 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978269    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-podinfo\") pod \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") "
Sep 18 16:06:51.978338 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978304    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-workload-socket\") pod \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") "
Sep 18 16:06:51.978338 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978331    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp5jw\" (UniqueName: \"kubernetes.io/projected/af9fcf83-d452-4650-bbcd-a5d13a7e8884-kube-api-access-sp5jw\") pod \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") "
Sep 18 16:06:51.978668 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978388    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-token\") pod \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\" (UID: \"af9fcf83-d452-4650-bbcd-a5d13a7e8884\") "
Sep 18 16:06:51.978668 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978423    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-data" (OuterVolumeSpecName: "istio-data") pod "af9fcf83-d452-4650-bbcd-a5d13a7e8884" (UID: "af9fcf83-d452-4650-bbcd-a5d13a7e8884"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:06:51.978668 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978498    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af9fcf83-d452-4650-bbcd-a5d13a7e8884" (UID: "af9fcf83-d452-4650-bbcd-a5d13a7e8884"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:06:51.978668 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978628    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af9fcf83-d452-4650-bbcd-a5d13a7e8884" (UID: "af9fcf83-d452-4650-bbcd-a5d13a7e8884"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:06:51.978668 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978646    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af9fcf83-d452-4650-bbcd-a5d13a7e8884" (UID: "af9fcf83-d452-4650-bbcd-a5d13a7e8884"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:06:51.979071 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978690    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-data\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:51.979071 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978690    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af9fcf83-d452-4650-bbcd-a5d13a7e8884" (UID: "af9fcf83-d452-4650-bbcd-a5d13a7e8884"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:06:51.979071 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978706    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-credential-socket\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:51.979071 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.978721    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-workload-socket\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:51.980657 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.980627    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af9fcf83-d452-4650-bbcd-a5d13a7e8884" (UID: "af9fcf83-d452-4650-bbcd-a5d13a7e8884"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:06:51.980751 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.980648    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af9fcf83-d452-4650-bbcd-a5d13a7e8884" (UID: "af9fcf83-d452-4650-bbcd-a5d13a7e8884"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:06:51.980751 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.980673    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9fcf83-d452-4650-bbcd-a5d13a7e8884-kube-api-access-sp5jw" (OuterVolumeSpecName: "kube-api-access-sp5jw") pod "af9fcf83-d452-4650-bbcd-a5d13a7e8884" (UID: "af9fcf83-d452-4650-bbcd-a5d13a7e8884"). InnerVolumeSpecName "kube-api-access-sp5jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:06:51.980751 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:51.980703    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-token" (OuterVolumeSpecName: "istio-token") pod "af9fcf83-d452-4650-bbcd-a5d13a7e8884" (UID: "af9fcf83-d452-4650-bbcd-a5d13a7e8884"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:06:52.079776 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.079747    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-token\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:52.079776 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.079771    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istiod-ca-cert\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:52.079776 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.079780    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-workload-certs\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:52.080031 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.079788    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-envoy\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:52.080031 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.079796    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9fcf83-d452-4650-bbcd-a5d13a7e8884-istio-podinfo\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:52.080031 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.079803    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp5jw\" (UniqueName: \"kubernetes.io/projected/af9fcf83-d452-4650-bbcd-a5d13a7e8884-kube-api-access-sp5jw\") on node \"ip-10-0-130-223.ec2.internal\" DevicePath \"\""
Sep 18 16:06:52.422417 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.422335    2559 generic.go:356] "Generic (PLEG): container finished" podID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerID="b792b8d3e9bf44350a7c12f5f51df2affba39cb7dcd628d2def381831918a4db" exitCode=0
Sep 18 16:06:52.422417 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.422380    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"
Sep 18 16:06:52.422589 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.422379    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" event={"ID":"af9fcf83-d452-4650-bbcd-a5d13a7e8884","Type":"ContainerDied","Data":"b792b8d3e9bf44350a7c12f5f51df2affba39cb7dcd628d2def381831918a4db"}
Sep 18 16:06:52.422589 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.422501    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj" event={"ID":"af9fcf83-d452-4650-bbcd-a5d13a7e8884","Type":"ContainerDied","Data":"7848257d75b9fc36ca31f5b476c4231ecc753841b31a7ffe6d518fd933e4fdd7"}
Sep 18 16:06:52.422589 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.422531    2559 scope.go:117] "RemoveContainer" containerID="b792b8d3e9bf44350a7c12f5f51df2affba39cb7dcd628d2def381831918a4db"
Sep 18 16:06:52.432004 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.431968    2559 scope.go:117] "RemoveContainer" containerID="66398535e44629b3fb3c976c20ade87e2d358d896e67d1587b556c3cd4d51053"
Sep 18 16:06:52.439362 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.439346    2559 scope.go:117] "RemoveContainer" containerID="b792b8d3e9bf44350a7c12f5f51df2affba39cb7dcd628d2def381831918a4db"
Sep 18 16:06:52.439607 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:06:52.439587    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b792b8d3e9bf44350a7c12f5f51df2affba39cb7dcd628d2def381831918a4db\": container with ID starting with b792b8d3e9bf44350a7c12f5f51df2affba39cb7dcd628d2def381831918a4db not found: ID does not exist" containerID="b792b8d3e9bf44350a7c12f5f51df2affba39cb7dcd628d2def381831918a4db"
Sep 18 16:06:52.439656 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.439615    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b792b8d3e9bf44350a7c12f5f51df2affba39cb7dcd628d2def381831918a4db"} err="failed to get container status \"b792b8d3e9bf44350a7c12f5f51df2affba39cb7dcd628d2def381831918a4db\": rpc error: code = NotFound desc = could not find container \"b792b8d3e9bf44350a7c12f5f51df2affba39cb7dcd628d2def381831918a4db\": container with ID starting with b792b8d3e9bf44350a7c12f5f51df2affba39cb7dcd628d2def381831918a4db not found: ID does not exist"
Sep 18 16:06:52.439656 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.439632    2559 scope.go:117] "RemoveContainer" containerID="66398535e44629b3fb3c976c20ade87e2d358d896e67d1587b556c3cd4d51053"
Sep 18 16:06:52.439921 ip-10-0-130-223 kubenswrapper[2559]: E0918 16:06:52.439893    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66398535e44629b3fb3c976c20ade87e2d358d896e67d1587b556c3cd4d51053\": container with ID starting with 66398535e44629b3fb3c976c20ade87e2d358d896e67d1587b556c3cd4d51053 not found: ID does not exist" containerID="66398535e44629b3fb3c976c20ade87e2d358d896e67d1587b556c3cd4d51053"
Sep 18 16:06:52.439982 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.439924    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66398535e44629b3fb3c976c20ade87e2d358d896e67d1587b556c3cd4d51053"} err="failed to get container status \"66398535e44629b3fb3c976c20ade87e2d358d896e67d1587b556c3cd4d51053\": rpc error: code = NotFound desc = could not find container \"66398535e44629b3fb3c976c20ade87e2d358d896e67d1587b556c3cd4d51053\": container with ID starting with 66398535e44629b3fb3c976c20ade87e2d358d896e67d1587b556c3cd4d51053 not found: ID does not exist"
Sep 18 16:06:52.446583 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.446548    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"]
Sep 18 16:06:52.448810 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:52.448791    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4e81aa-openshift-default-579787bfd4-tcflj"]
Sep 18 16:06:53.743117 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:06:53.743081    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" path="/var/lib/kubelet/pods/af9fcf83-d452-4650-bbcd-a5d13a7e8884/volumes"
Sep 18 16:07:40.920758 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:07:40.920722    2559 scope.go:117] "RemoveContainer" containerID="bd441383c96cf5e7ca0de6d1c94a77080ad301a8a71e8c4343c4cbc8128e769c"
Sep 18 16:07:40.929000 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:07:40.928980    2559 scope.go:117] "RemoveContainer" containerID="cf1936d15adab7952f3645d260bebb6d93502008ac7b63a3a2efe3c1b0298b90"
Sep 18 16:07:40.936467 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:07:40.936443    2559 scope.go:117] "RemoveContainer" containerID="8982b22dcdc04f0dd269cec48dca1692ad381f7cc3a39f7e0c7e1b03141861a6"
Sep 18 16:08:09.978171 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:08:09.978140    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:08:10.274786 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:08:10.274699    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:08:11.673618 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:08:11.673581    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:08:29.947048 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:08:29.947010    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:08:30.047790 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:08:30.047754    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:08:33.985581 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:08:33.985544    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:08:42.473497 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:08:42.473442    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:08:51.076135 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:08:51.076102    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:08:58.673817 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:08:58.673783    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:09:08.387119 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:09:08.387082    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:09:18.169964 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:09:18.169922    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:09:42.169541 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:09:42.169501    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:09:42.234749 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:09:42.234716    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:10:05.815045 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:05.815004    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:10:05.886125 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:05.886094    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:10:24.534451 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534373    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj"]
Sep 18 16:10:24.534831 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534815    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b261b9f5-63ab-459e-894d-d8d9898c95f0" containerName="payload-pre-processing"
Sep 18 16:10:24.534912 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534834    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b261b9f5-63ab-459e-894d-d8d9898c95f0" containerName="payload-pre-processing"
Sep 18 16:10:24.534912 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534850    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerName="istio-proxy"
Sep 18 16:10:24.534912 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534879    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerName="istio-proxy"
Sep 18 16:10:24.534912 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534887    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0dc1d4b-70a4-4029-a683-5dd9716b4dee" containerName="revoke-keys"
Sep 18 16:10:24.534912 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534895    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0dc1d4b-70a4-4029-a683-5dd9716b4dee" containerName="revoke-keys"
Sep 18 16:10:24.534912 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534912    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82afb053-d882-4da8-8544-bea819831570" containerName="payload-processing"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534921    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="82afb053-d882-4da8-8544-bea819831570" containerName="payload-processing"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534934    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerName="istio-proxy"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534939    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerName="istio-proxy"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534954    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fca1c4b5-d927-4570-a16c-93de369c34f2" containerName="payload-pre-processing"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534959    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fca1c4b5-d927-4570-a16c-93de369c34f2" containerName="payload-pre-processing"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534966    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199bef39-11ee-4aca-bd23-778459f6b5e0" containerName="payload-processing"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.534971    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="199bef39-11ee-4aca-bd23-778459f6b5e0" containerName="payload-processing"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.535051    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerName="istio-proxy"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.535058    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="82afb053-d882-4da8-8544-bea819831570" containerName="payload-processing"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.535065    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fca1c4b5-d927-4570-a16c-93de369c34f2" containerName="payload-pre-processing"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.535072    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b261b9f5-63ab-459e-894d-d8d9898c95f0" containerName="payload-pre-processing"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.535077    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="af9fcf83-d452-4650-bbcd-a5d13a7e8884" containerName="istio-proxy"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.535083    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="199bef39-11ee-4aca-bd23-778459f6b5e0" containerName="payload-processing"
Sep 18 16:10:24.535113 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.535088    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0dc1d4b-70a4-4029-a683-5dd9716b4dee" containerName="revoke-keys"
Sep 18 16:10:24.537835 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.537790    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj"
Sep 18 16:10:24.544684 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.544664    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hq9sc\""
Sep 18 16:10:24.550634 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.550609    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn4jn\" (UniqueName: \"kubernetes.io/projected/df4a6b7a-22aa-4852-926c-ae09a3d2a4c6-kube-api-access-zn4jn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rchhj\" (UID: \"df4a6b7a-22aa-4852-926c-ae09a3d2a4c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj"
Sep 18 16:10:24.550781 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.550706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df4a6b7a-22aa-4852-926c-ae09a3d2a4c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rchhj\" (UID: \"df4a6b7a-22aa-4852-926c-ae09a3d2a4c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj"
Sep 18 16:10:24.552168 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.552142    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj"]
Sep 18 16:10:24.651124 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.651084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df4a6b7a-22aa-4852-926c-ae09a3d2a4c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rchhj\" (UID: \"df4a6b7a-22aa-4852-926c-ae09a3d2a4c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj"
Sep 18 16:10:24.651302 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.651151    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn4jn\" (UniqueName: \"kubernetes.io/projected/df4a6b7a-22aa-4852-926c-ae09a3d2a4c6-kube-api-access-zn4jn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rchhj\" (UID: \"df4a6b7a-22aa-4852-926c-ae09a3d2a4c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj"
Sep 18 16:10:24.651469 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.651448    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df4a6b7a-22aa-4852-926c-ae09a3d2a4c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rchhj\" (UID: \"df4a6b7a-22aa-4852-926c-ae09a3d2a4c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj"
Sep 18 16:10:24.661560 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.661537    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn4jn\" (UniqueName: \"kubernetes.io/projected/df4a6b7a-22aa-4852-926c-ae09a3d2a4c6-kube-api-access-zn4jn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rchhj\" (UID: \"df4a6b7a-22aa-4852-926c-ae09a3d2a4c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj"
Sep 18 16:10:24.848264 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:24.848182    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj"
Sep 18 16:10:25.131377 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:25.131294    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj"]
Sep 18 16:10:25.135099 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:10:25.135073    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4a6b7a_22aa_4852_926c_ae09a3d2a4c6.slice/crio-b7829b7a5db4e6a1ba0f5dd5f0a4ac5cbbe8ac32466aa46098aa19a22dafef74 WatchSource:0}: Error finding container b7829b7a5db4e6a1ba0f5dd5f0a4ac5cbbe8ac32466aa46098aa19a22dafef74: Status 404 returned error can't find the container with id b7829b7a5db4e6a1ba0f5dd5f0a4ac5cbbe8ac32466aa46098aa19a22dafef74
Sep 18 16:10:25.421428 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:25.421333    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj" event={"ID":"df4a6b7a-22aa-4852-926c-ae09a3d2a4c6","Type":"ContainerStarted","Data":"137ed6f4632d4bc55f04a00821c3e94682331808a27c7a991f138eb708fa4ab9"}
Sep 18 16:10:25.421570 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:25.421531    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj"
Sep 18 16:10:25.421612 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:25.421578    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj" event={"ID":"df4a6b7a-22aa-4852-926c-ae09a3d2a4c6","Type":"ContainerStarted","Data":"b7829b7a5db4e6a1ba0f5dd5f0a4ac5cbbe8ac32466aa46098aa19a22dafef74"}
Sep 18 16:10:25.446529 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:25.446481    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj" podStartSLOduration=1.446468843 podStartE2EDuration="1.446468843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:10:25.443895427 +0000 UTC m=+2148.005127568" watchObservedRunningTime="2026-09-18 16:10:25.446468843 +0000 UTC m=+2148.007700982"
Sep 18 16:10:36.429331 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:36.429296    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rchhj"
Sep 18 16:10:44.933083 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:44.933050    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:10:51.925342 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:51.925301    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:10:59.625748 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:10:59.625707    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:11:02.268388 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:11:02.268335    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:11:04.530679 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:11:04.530638    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:11:08.840427 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:11:08.840385    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:11:12.843028 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:11:12.842994    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:11:37.011586 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:11:37.011543    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:11:37.105397 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:11:37.105356    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:11:40.983362 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:11:40.983332    2559 scope.go:117] "RemoveContainer" containerID="71994f07b0ce015ff68217f991851a2d0d6731ebbc2ea78eafa05be6dcb3fba7"
Sep 18 16:11:40.991541 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:11:40.991520    2559 scope.go:117] "RemoveContainer" containerID="3e9615e7d2c766c13f46eb6e7ac8b80da709c27fbbcafb8e2687b677ad020ead"
Sep 18 16:11:55.921585 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:11:55.921504    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:11:57.099710 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:11:57.099675    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:11:59.138523 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:11:59.138486    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:12:07.544961 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:07.544921    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:12:15.829275 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:15.829234    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:12:22.636930 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:22.636886    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:12:29.844610 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:29.844569    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:12:38.443686 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:38.443645    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:12:41.012200 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:41.012168    2559 scope.go:117] "RemoveContainer" containerID="7085d3882def6c68c4ffc67498b3d0aeeb5102b9ff8fc86c1ee40a85c43d4740"
Sep 18 16:12:41.020222 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:41.020203    2559 scope.go:117] "RemoveContainer" containerID="70e2954834f396a6b06b60655f8fd7474438a937ab8fd9b1b711191934d8cff5"
Sep 18 16:12:45.932149 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:45.932107    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rlx5z"]
Sep 18 16:12:49.569247 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:49.569206    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:12:49.663708 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:49.663666    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:12:52.757409 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:52.757384    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dzwbg_ac1a1089-9a91-4f0c-a285-79e15b4da235/global-pull-secret-syncer/0.log"
Sep 18 16:12:52.970738 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:52.970709    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log"
Sep 18 16:12:53.265450 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:53.265423    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-223.ec2.internal_2b01950708f7973287a6e4968c09a493/haproxy/0.log"
Sep 18 16:12:56.377312 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:56.377286    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-zqqmm_2f1e38d2-8012-49ce-bdc4-de7d55d11239/manager/0.log"
Sep 18 16:12:56.411767 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:56.411743    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-lgxst_a3f00e9c-d13a-4e64-acb1-f8c5754697c5/manager/0.log"
Sep 18 16:12:56.460583 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:56.460545    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-57xz8_7e46a8d9-f7b7-469f-99d7-1a13f77741b9/konnectivity-agent/0.log" limit=1024
Sep 18 16:12:56.491946 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:56.491917    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-rchhj_df4a6b7a-22aa-4852-926c-ae09a3d2a4c6/manager/0.log"
Sep 18 16:12:56.534967 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:56.534936    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-rlx5z_b5947521-edec-444b-abe3-33c010639ce1/limitador/0.log"
Sep 18 16:12:58.793163 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:58.793070    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-6wqjh_2bc4f75b-bab6-4f04-9848-f6ff1dd19297/kube-state-metrics/0.log"
Sep 18 16:12:58.813045 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:58.813020    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-6wqjh_2bc4f75b-bab6-4f04-9848-f6ff1dd19297/kube-rbac-proxy-main/0.log"
Sep 18 16:12:58.837923 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:58.837897    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-6wqjh_2bc4f75b-bab6-4f04-9848-f6ff1dd19297/kube-rbac-proxy-self/0.log"
Sep 18 16:12:59.029438 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:59.029406    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9knb9_ab551dde-5363-4914-bb99-c0e65d9ce995/node-exporter/0.log"
Sep 18 16:12:59.055080 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:59.055015    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9knb9_ab551dde-5363-4914-bb99-c0e65d9ce995/kube-rbac-proxy/0.log"
Sep 18 16:12:59.080297 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:59.080281    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9knb9_ab551dde-5363-4914-bb99-c0e65d9ce995/init-textfile/0.log"
Sep 18 16:12:59.335196 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:59.335096    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a32551d5-e289-49b2-a7fd-f584a56ac5b2/prometheus/0.log"
Sep 18 16:12:59.361196 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:59.361171    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a32551d5-e289-49b2-a7fd-f584a56ac5b2/config-reloader/0.log"
Sep 18 16:12:59.397187 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:59.397165    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a32551d5-e289-49b2-a7fd-f584a56ac5b2/thanos-sidecar/0.log"
Sep 18 16:12:59.424382 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:59.424356    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a32551d5-e289-49b2-a7fd-f584a56ac5b2/kube-rbac-proxy-web/0.log"
Sep 18 16:12:59.454440 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:59.454418    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a32551d5-e289-49b2-a7fd-f584a56ac5b2/kube-rbac-proxy/0.log"
Sep 18 16:12:59.480773 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:59.480753    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a32551d5-e289-49b2-a7fd-f584a56ac5b2/kube-rbac-proxy-thanos/0.log"
Sep 18 16:12:59.510212 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:12:59.510193    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a32551d5-e289-49b2-a7fd-f584a56ac5b2/init-config-reloader/0.log"
Sep 18 16:13:01.466314 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:01.466264    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-d2g64_2dcd3e26-1fc0-4356-80a4-fbe8a510822c/networking-console-plugin/0.log"
Sep 18 16:13:03.179228 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.179189    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"]
Sep 18 16:13:03.184352 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.184329    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.187200 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.187181    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2zs2w\"/\"default-dockercfg-k85dh\""
Sep 18 16:13:03.187366 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.187231    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2zs2w\"/\"kube-root-ca.crt\""
Sep 18 16:13:03.188533 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.188511    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2zs2w\"/\"openshift-service-ca.crt\""
Sep 18 16:13:03.192398 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.192372    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"]
Sep 18 16:13:03.300163 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.300128    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6nw\" (UniqueName: \"kubernetes.io/projected/b9128862-f667-4168-9f3c-136b81742b6b-kube-api-access-nt6nw\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.300163 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.300167    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9128862-f667-4168-9f3c-136b81742b6b-lib-modules\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.300383 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.300190    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b9128862-f667-4168-9f3c-136b81742b6b-proc\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.300383 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.300295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b9128862-f667-4168-9f3c-136b81742b6b-podres\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.300383 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.300325    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9128862-f667-4168-9f3c-136b81742b6b-sys\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.400902 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.400849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b9128862-f667-4168-9f3c-136b81742b6b-podres\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.401087 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.400914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9128862-f667-4168-9f3c-136b81742b6b-sys\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.401087 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.400959    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6nw\" (UniqueName: \"kubernetes.io/projected/b9128862-f667-4168-9f3c-136b81742b6b-kube-api-access-nt6nw\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.401087 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.400990    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9128862-f667-4168-9f3c-136b81742b6b-lib-modules\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.401087 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.400995    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9128862-f667-4168-9f3c-136b81742b6b-sys\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.401087 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.401015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b9128862-f667-4168-9f3c-136b81742b6b-proc\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.401087 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.401017    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b9128862-f667-4168-9f3c-136b81742b6b-podres\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.401087 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.401066    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b9128862-f667-4168-9f3c-136b81742b6b-proc\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.401333 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.401122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9128862-f667-4168-9f3c-136b81742b6b-lib-modules\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.409737 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.409715    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6nw\" (UniqueName: \"kubernetes.io/projected/b9128862-f667-4168-9f3c-136b81742b6b-kube-api-access-nt6nw\") pod \"perf-node-gather-daemonset-9zzms\" (UID: \"b9128862-f667-4168-9f3c-136b81742b6b\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.494869 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.494819    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:03.767292 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:03.767265    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"]
Sep 18 16:13:03.769166 ip-10-0-130-223 kubenswrapper[2559]: W0918 16:13:03.769131    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb9128862_f667_4168_9f3c_136b81742b6b.slice/crio-fb68f8bbb3b708eccf7b39e4f0aaa8282a60db61867401310beed93196581526 WatchSource:0}: Error finding container fb68f8bbb3b708eccf7b39e4f0aaa8282a60db61867401310beed93196581526: Status 404 returned error can't find the container with id fb68f8bbb3b708eccf7b39e4f0aaa8282a60db61867401310beed93196581526
Sep 18 16:13:04.178967 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:04.178850    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms" event={"ID":"b9128862-f667-4168-9f3c-136b81742b6b","Type":"ContainerStarted","Data":"477e88e0f3bd39894e18e00a1f94d5e2969787dcdc078211064b535821b2aaa9"}
Sep 18 16:13:04.178967 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:04.178939    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:04.178967 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:04.178961    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms" event={"ID":"b9128862-f667-4168-9f3c-136b81742b6b","Type":"ContainerStarted","Data":"fb68f8bbb3b708eccf7b39e4f0aaa8282a60db61867401310beed93196581526"}
Sep 18 16:13:04.198581 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:04.198528    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms" podStartSLOduration=1.198515616 podStartE2EDuration="1.198515616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:13:04.196723803 +0000 UTC m=+2306.757955942" watchObservedRunningTime="2026-09-18 16:13:04.198515616 +0000 UTC m=+2306.759747757"
Sep 18 16:13:04.839169 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:04.839123    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-87jdl_0e563fb7-63b4-47f8-9c20-5dd15ba96472/dns/0.log"
Sep 18 16:13:04.869689 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:04.869665    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-87jdl_0e563fb7-63b4-47f8-9c20-5dd15ba96472/kube-rbac-proxy/0.log"
Sep 18 16:13:05.029101 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:05.029070    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fpfzs_792d0ad5-2007-42a0-bb50-a3b4c0ef1af5/dns-node-resolver/0.log"
Sep 18 16:13:05.664157 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:05.664130    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f987fb898-ndqst_5f16eb75-a7c7-4904-9b96-a71c1b45dcd7/registry/0.log"
Sep 18 16:13:05.749774 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:05.749749    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nrqvf_51596704-4c57-46c5-a82a-db2ad011ee51/node-ca/0.log"
Sep 18 16:13:07.257454 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:07.257419    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-z8d9v_30914206-a17b-456a-89cc-decc238d701f/payload-pre-processing/0.log"
Sep 18 16:13:07.283937 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:07.283914    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-mx9mf_619761f1-b0a4-4b32-978d-5c3702133c7d/payload-processing/0.log"
Sep 18 16:13:08.314244 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:08.314213    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dw5xs_f60485ae-76c5-4303-86a2-749c71003198/serve-healthcheck-canary/0.log"
Sep 18 16:13:08.903913 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:08.903884    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4jk9j_b4d75691-4e93-4377-8729-37d2b04c2087/kube-rbac-proxy/0.log"
Sep 18 16:13:08.949920 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:08.949843    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4jk9j_b4d75691-4e93-4377-8729-37d2b04c2087/exporter/0.log"
Sep 18 16:13:08.995008 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:08.994981    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4jk9j_b4d75691-4e93-4377-8729-37d2b04c2087/extractor/0.log"
Sep 18 16:13:10.192955 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:10.192924    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-9zzms"
Sep 18 16:13:11.425442 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:11.425409    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-65b57fbbf4-wj42c_a029d48e-b804-490a-8f0b-26da978a9a9a/manager/0.log"
Sep 18 16:13:11.452645 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:11.452619    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-n5q6z_0fdb8fa7-270e-45a4-8ee5-1a0a4b6c2302/manager/0.log"
Sep 18 16:13:11.847240 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:11.847209    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hbfxb_92275e0d-82a3-472b-88ca-d630ba4f36da/manager/0.log"
Sep 18 16:13:12.295097 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:12.295062    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54b8c8d665-pm8h9_3e64e9a8-1885-4a01-a361-241cbe4b96d2/manager/0.log"
Sep 18 16:13:12.313087 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:12.313060    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54b8c8d665-pm8h9_3e64e9a8-1885-4a01-a361-241cbe4b96d2/copy-manifests/0.log"
Sep 18 16:13:13.594337 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:13.594310    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-z8d9v_30914206-a17b-456a-89cc-decc238d701f/payload-pre-processing/0.log"
Sep 18 16:13:13.610036 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:13.610013    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-mx9mf_619761f1-b0a4-4b32-978d-5c3702133c7d/payload-processing/0.log"
Sep 18 16:13:13.829470 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:13.829427    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-c59846bb8-qwrhp_21c9275d-e252-4185-91db-aceedf69e3a0/manager/0.log"
Sep 18 16:13:15.753743 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:15.753710    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-z8d9v_30914206-a17b-456a-89cc-decc238d701f/payload-pre-processing/0.log"
Sep 18 16:13:15.771720 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:15.771690    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-mx9mf_619761f1-b0a4-4b32-978d-5c3702133c7d/payload-processing/0.log"
Sep 18 16:13:19.066881 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:19.066824    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-4l5h5_019d191c-04e0-40df-bfe7-e3c5e53c874d/migrator/0.log"
Sep 18 16:13:19.090431 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:19.090408    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-4l5h5_019d191c-04e0-40df-bfe7-e3c5e53c874d/graceful-termination/0.log"
Sep 18 16:13:20.588130 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:20.588100    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6bnzx_60b93ec7-8e32-4d00-a695-ca96c49a0609/kube-multus/0.log"
Sep 18 16:13:20.824895 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:20.824869    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lnfcs_9085c12e-916c-423c-a2da-8ba78ec5fc0c/kube-multus-additional-cni-plugins/0.log"
Sep 18 16:13:20.850129 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:20.850035    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lnfcs_9085c12e-916c-423c-a2da-8ba78ec5fc0c/egress-router-binary-copy/0.log"
Sep 18 16:13:20.875044 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:20.875018    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lnfcs_9085c12e-916c-423c-a2da-8ba78ec5fc0c/cni-plugins/0.log"
Sep 18 16:13:20.898728 ip-10-0-130-223 kubenswrapper[2559]: I0918 16:13:20.898708    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lnfcs_9085c12e-916c-423c-a2da-8ba78ec5fc0c/bond-cni-plugin/0.log"